Inside the Change at FIS: A Global Enterprise's Road to DevOps Mastery

All right, well, good afternoon, everyone, and welcome.

We appreciate you coming, and I want to thank the DevOps Summit and IT Revolution for giving us this opportunity to tell our story about global transformation to DevSecOps. Although many people still cringe when we use that, I think it makes sense because we really are embedding security into our DevSecOps pipeline.

Bob and I are going to tell you a little bit about our story. We're going to start by first telling you a little bit of who we are.

I'm with Fidelity Information Services, a company that's been around for 50 years and has been through many acquisitions. In fact, if you go look up our history online, the one thing you'll see is a graph or a line chart of all of our acquisitions. Fifteen acquisitions over the last 50 years. We have about 55,000 people across the globe, and we're at 241 in the Fortune 500.

We are what many people call a FinTech, financial technology company. We produce banking cores, payment processing under our Worldpay branch, and wealth management services. Companies unlike JPMorgan Chase, who can build their own stuff, banks that can't, we build it for them. Many times, we run their bank and back end for them.

I'm part of the banking sector. We also, like I said, have wealth management, and we also have the big Worldpay acquisition we did, which runs as a separate unit for all the payment processing.

So Bob, a little bit of Opus?

Yep, thank you.

Opus Technologies is a 26-year-old technology services firm. We're focused primarily in the banking, FinTech, and payments space. That's our lane, if you will. And we are focused on transforming and modernizing platforms within that sector.

We have particular skills in DevSecOps, API management, cloud initiatives, and data analytics. So I'm here to join Dan on this endeavor here.

So a little bit about our background. I would suspect our background isn't a lot different than many of your backgrounds, but we grew through acquisitions. I love the theme of this conference, right? Together, to go faster. And you'll see, even in our little logo, you'll see "pivot together," because what's interesting through all these acquisitions, when you first meet someone at FIS, they'll tell you what acquisition they came from. It's almost going to happen within that conversation, right?

So the culture becomes a big deal. "I'm from this culture." Well, aren't you part of FIS's culture as well? And that's really going to be a big theme in our story, is that these types of transitions are really cultural based.

These acquisitions have made that more difficult because not only did we acquire a different culture, we acquired a different technology platform at different states of maturity, different processes, and ways of working. So those all came together through 15 big acquisitions that brought a lot of big software.

There's a lot of redundancy across that portfolio, a lot of systems that do the same thing but do it a bit differently. We have efforts underway, like many of you, to modernize our technology stack and try to move those forward, but they move very slow.

Most of our platforms are very monolithic. A lot of redundant capabilities, limited services. Our API frameworks, even though we use a lot of them, we don't have good API frameworks with all of the kind of capabilities you'd want for testing without having to involve the other party. So that makes it very difficult.

And then we have a lot of ticket-based manual processes, which I'm sure many of you are familiar with, but this adds a lot of toil to our developers and engineers, who are constantly opening, chasing, and managing tickets most of the time.

This has led to the challenges we face: long delivery times, we're slow to innovate, we have a hard time onboarding new customers. It can take years to onboard a new banking customer, and that, the banks don't care for. Nobody likes to go through it.

We saw on a slide, I think it was at the testing one I went to a little earlier, that more and more of your time, as these things grow older, is spent on maintaining and just dealing with technical debt, leaving very little time to actually work on innovation and new products, new features.

These older platforms require a lot of security challenges. Yesterday, we just started a thing to go through and find all the problems with Spring. We have about 25 versions running, and now we're telling them, "We've got to fix them, right? They're not going to be supported anymore." So that's something completely out of the blue we've got to deal with. Of course, it's because there's a security threat in those older ones.

And then you have talent challenges because a lot of people don't know or even want to work on the older platforms. Anyone who's ever tried to move some legacy platform offshore knows that it's hard to get people who want to do COBOL development. Can you blame them?

Testing is predominantly manual, and we have a lack of standardized metrics that we can measure maturity or how well we're doing across teams, so we can get a fair assessment of how they're doing.

But our customers, they don't care about any of that. And I'm sure yours don't either, right? What they want, they want to consume things as a service. They want it to be easy, and they want a rapid pace of new features and capabilities. They want to customize with unconstrained customization. They want to be able to fit it in.

In the banking world in particular, or the finance world, everything's becoming embedded. I mean, you buy a car now, the whole loan process is built into buying the car, right? It's almost in an app, in fact. And you're seeing more and more of that, that the banks, which you're still going to have, are becoming part of an ecosystem, a financial ecosystem where a lot of the financial transactions are embedded in the process.

They're no longer, "Oh, I have to go to my bank and get that loan, and then I'll come back and talk to you." It's like, "I'm buying the car. Oh, here's the loan process. Let's just click through that."

So that embedded piece means all those banking systems that used to work independently all have to work now in this ecosystem. And they have to do it in real time. They have to do it securely. They have to check for fraud, they have to check for money laundering, they have to do all that on every single transaction, no matter where it happens in the world, and deal with transferring the currencies and all those other things.

And it has to be done fair, secure, and reliable, but it can't break, right? Because nobody wants their money screwed up. So that makes it very difficult to keep all these things working. And nobody wants to touch anything, right? Because, my God, it'd break it if we touch it. So just leave it alone. It's working.

So we started on our journey. We call it Operation Joined Arrows. It came out of one of the former leaders. He was a big military guy, and Operation Joined Arrows is what we got. But it's a nice theme because it brings us together, and it's really about working together.

We didn't want this just to be a technology transformation. We really wanted to talk about how can we transform the culture? How can we do things differently than the way we're doing?

I have an old saying, that culture's like a rubber band. You can stretch it, but it just wants to go back the way it was. And really, until you break that rubber band, you really can't change your culture.

We started looking at what are the cultural elements of our transformation that we're going to have to deal with. We knew we needed to really optimize the developer experience. It isn't good, right? Developers are not happy.

We started doing surveys. I actually got a team of DevX people. That's all they work on, is trying to eliminate toil and finding out where those toils are. We do quarterly surveys to see how we're doing and the perception of our engineering staff. Do they feel like we're getting better? That's a big measure for us. Are we improving developer experience? Happier developers develop better code, they get more done, they're more productive. All the literature supports this.

We really wanted to promote, and I know this is an overused term, the shift left. But really, this is about getting those things earlier in the process, especially unit testing and all the other testing and automating that, performance testing, security compliance, earliest possible. Because we all know catching it early costs a lot less and it takes a lot less time than catching it later. I don't need to drill you guys on that.

Data-driven. We really wanted to move away from, "I think this works best." I think maybe it was JPMorgan Chase talking a little bit about this. We don't want opinionated, "This is the best way to get things done." We want to work on what works using evidence.

When we do something and it has an impact, and it's having the impact we expect, we're going to keep doing that and we're going to put more money towards that. When we do something and we don't see much of an impact, we're not going to put as much effort into that. Now, some of these things are just sort of have-to-dos, and we'll deal with those. But really, we want to focus our effort where we see results.

We want to really get to that role model, right? Build, run, operate. A team that's responsible for building and operating a platform, if they're not empowered to do that because they have to go through 14 other teams to get one decision made or one thing moved to production, and open up a zillion tickets to people they have no idea who they are, they're not empowered.

So if you want to move to build-run-operate, we've got to empower the teams, with the end state being continuous delivery, which is really all about dramatically improving, accelerating our ability to adopt an agile culture, to empower teams to continue to deliver high-quality, secure software at scale. We want to lower the cost of ownership, and at the same time, we want to improve through our net promoter score. We actually want to see improvements in scores over the client satisfaction and experience.

So there's a lot in this slide. It's a lot to unpack. It's kind of hard to talk to.

We've been on the DevOps journey for a long time, and the CI/CD pipeline journey, or whatever you want to call it. It was always called CI/CD pipeline for a long time. It was even agile a little bit. We had many failed attempts, three big ones, in fact, where we tried to go and do this.

And so when we started this time, we said, "You know, I think the problem has been," in talking to a lot of teams and looking at what we did, "we tried to do too much at once." We have 50 strategic platforms across the banking sector. These platforms are very large. Some of these teams have hundreds and hundreds of people on them. It was too much for them to take on a full DevSecOps transformation at once, and we weren't ready to take them all on at once.

So we said, "Let's break the problem up. Let's break it up into pieces." Hadn't seen anyone else do this, but we thought, "Why not? Let's do it."

So we broke the first phase up, and we did continuous integration. We said, "Let's get everybody onto the same repo, right? Let's get them all onto Bitbucket. Let's get everybody using Jenkins. Let's get everybody using Artifactory. Let's get everybody onto the same set of tools as a starting point, just to get things going."

And then, while we're working with each one of those platform teams, let's start teaching them about the future. Let's teach them what is trunk-based branching. How can they get to it? Why is it important? Why should they care? Let's talk to them about other enablers or accelerators that are going to allow them to eventually one day do continuous delivery, such as continuous testing, such as continuous compliance, automation. I call it governance as a service. How do we get to those points?

We didn't actually do all those in the first phase, but we did get those three big transformations done, right? And that was a pretty big thing. That took us about nine months to do 48 platforms, affecting over 3,000 people in teams.

And next we said, "All right, now let's get ready for our continuous delivery pilot." And this one, we weren't in as good a shape because our technology CTO team really didn't have tools picked yet, and everybody was kind of doing their own thing. We had quite a few different tools out there for doing continuous delivery, and most people were doing it manually.

So this one, we decided to do a pilot because we weren't quite sure what direction we were going to go. And there was some technology we needed to make decisions on. So we ended up using one of our big platform teams. They had worked with a tool called Harness, which we ended up adopting because it provided a high level of automation and orchestration for our continuous delivery processes.

And then we started connecting it into everything. Connecting it into ServiceNow, connecting it into all of our software testing tools, connecting it into our governance compliance framework tools, connecting it into an endless list of tools. As I'll show you before, there's a lot of tools involved in this. I'm always sort of shocked at how many tools it takes to make this work.

And then that took about another nine months. We just finished that, and you've caught us just as we're going into phase three, which is now to take what we did in phase two and scale it across the other 48 platforms. As you can imagine, I'm a bit nervous about it because expectations are very high, because we have some pretty good results out of phase two.

But we had to work, as you can see in this slide, with many other people. And that's a key thing to this, right? Again, looking at the many different teams within our organization, there isn't just one initiative underway. There's an agile initiative, there's my DevSecOps initiative, there's a continuous testing initiative. There's an SRE organization who's gearing up and starting to do quite a bit of shift-left activity. And there's an infrastructure-as-a-service initiative, which is really moving to get to a point where we can do API-based infrastructure for on-prem, not the cloud.

So there's a lot of coordination needed to go across this pipeline. All these different teams have a role to play. They all need to work together. And having a shared outcome that, "Hey, don't go buy a tool that requires us to put a manual process in place. Don't ask us to do a manual process. Remember, we're trying to get to a point where a team can do continuous delivery, where they can deliver in minutes if that's what they want, if that fits with their customers and their... If they want to do it in weeks, they want to do it in days, whatever."

But today, based on the process mapping we've done, we're seeing that that's taking anywhere from four to six months. So we've got a long ways to go.

I'm not going to spend a lot of time on these tools slides, but there's a lot of tools that we're integrating into the pipeline. And then this is the real piece of how we do the transformation that I think is really unique, that has made it work.

Starting out with the gap assessment, we try to make that very lightweight. We do a lot of the heavy lifting. We mine their current documents, then we do some sessions with them to understand. We put this all into our wiki and document the daylights out of it.

We then do value stream mapping to understand how long does it take them to get from the point of code commit to production, and every step they go through to get there. This is where we baseline their maturity and understand where they're at.

We do discovery meetings with them. By the way, that value stream mapping took a lot longer than we expected it would because we couldn't find any one person who understood their whole value stream. We had to talk to many different teams, and that was a surprise to us.

Discovery meetings, and then finally roadmap alignment. Agreeing after the discovery meeting, "Here's what we found. Here's what we're going to do. Here's what we're going to do to get you to the next level of maturity." Getting them to sign off and agree to it.

And this is where our team friends from Opus come in. They come in and do all the heavy lifting. We bring a whole team in for every single platform to do the transformation work for them, building their new pipelines, migrating their code, doing all the steps that would need to be done, training, getting them ready, understanding who needs to learn what, and then handing that off to them.

Of course, we document the daylights out of everything, and then we get a sign-off and turn it over to that team to run. Doing it this way, we've found, has been very successful, because where we couldn't get their time before to do a lot of that work, we hand off to Opus, who's going to talk a little bit about that process now, how we did that.

Well, here's where we go from here, right?

So if any of you were at Gene Kim's speech yesterday, or with Dr. Spear, he talked about the danger zone and then there's the safe zone. So we came in, and we came right into the danger zone. A lot of changes that were going on, the danger zone.

But we had a fantastic core group of DevSecOps engineers, subject experts, cloud team members, and so on, ready to go. That was nice, but we had to ramp up really quickly for this initiative.

If you think about the pandemic, the tail end of the pandemic, it was tough to get people to move a bit. We had to take them through the rigorous interviewing and background check process. I have to say, even the folks doing background checks were impacted by COVID, when they were with their teams trying to check those out.

So ultimately, we got it figured out and we started bringing people on very rapidly. And then we were also worried about other things like retention, things like rising resource costs, things that we had to manage. Dan didn't have to worry about that. That was a good move on his behalf.

But a lot of the things that you deal with every day in your organization, we were dealing with as a partner. The other thing we had to talk about, Dan talked about cultures, multiple cultures throughout FIS. Oftentimes, we were the tip of the spear on going into different platform, line-of-business leaders, tech leaders, and so on, and trying to understand those different cultures across the enterprise.

We think we were part of that change of shifting culture for FIS on Dan's behalf. A lot of things happened. There were change, CEO change, cost-cutting changes, things that we had to bear the brunt of for a short period of time, but we worked through all that.

So, pretty challenging times to get things going. But along the way, we made commitments, and our critical success factors here were our CEO, knowing that there'd been a couple of attempts at FIS to get this done before, he had to step up and say, "We're going to make this work for Dan. This is going to be successful no matter what."

So we put a lot of time and effort and resource behind that. We did a lot of reporting. We made sure that excellent reporting, good, bad, and the ugly, was surfaced very quickly. Some of you have heard of this in other sessions. I think that was very helpful.

We had to make sure we had early wins for Dan, to make sure his team and all were successful. We're getting a little short on time, but we did a lot of documentation. We did what Dan mentioned here, full transparency. And then we tried to bring experience that we've had working with other major banks, FinTechs, and payments organizations, things we've seen across the globe, really, with these large enterprise tech customers, and tried to help bring some of that value to Dan.

So Dan, thanks.

And we were very pleased, then, after the end of this. I want to put a plug in for Mike, Mark Hornbeek, who is our guru and DevSecOps expert that you brought in, Bob.

Yes.

And has really helped us understand what does it take, how to be careful. And he put us in for this award, which we were thrilled to win. We were surprised, as you can imagine.

But it's been very helpful, believe it or not, internally, bringing a level of competency, belief in what we're doing, and external validation that what we're doing is actually having an impact and really making a difference.

I think a lot of people inside the company who don't understand the complexities of adopting DevSecOps don't understand how difficult this is. This helped us get some of that credibility. That's the word I wanted. Because we ask for a lot of money to do this work, and getting that money has been a continual... That's pretty much like my full-time job, right? Getting money so I can keep this thing running.

But we're getting results, right? And I think that's really the important thing here. We reduced build times in the first phase by, on average, 70%. That was tremendous, right? People were pretty shocked when they started seeing those improvements in build times. And productivity went up about 25% as well, which means they can do a little bit more with less.

Now, in the second phase, we saw the lead time for changes dropped by 22%, and productivity only by 12%. And here's an important lesson out of phase two that you want to keep in mind. More mature teams don't get the benefit of just moving to an automated platform, because they're already doing it. They're just using different tools, or they've got a different way of doing it.

This was somewhat of a surprise to us. It shouldn't have been. So we had teams that were very immature, they got big boosts, 50% in some cases, right, when we moved them onto the automated platforms. Teams that were very mature, they saw almost no improvement, maybe a few percentage points, because we took out a few toil and a few things for them, but really not the kind of percentage improvement we wanted.

So that was a big learning lesson. And because of that, we have now, as we're going into phase three, 20 accelerators. And with those teams, we're going to have to do more. We're going to have to get them on continuous testing, or maybe we're going to have to do some re-architecting, or maybe they're going to have to be one of the first ones to go to governance as a service. Or maybe they're going to have to start using infrastructure as a service, right?

So we're going to have to do something else with those teams, because they're not just going to get a benefit just by moving to an automated pipeline.

And then finally, it's all about what do you measure, right? It's about the data. We wanted to be able to have a consistent way across the organization, and Mark led us to the DORA metrics. We looked at a bunch of metric frameworks. We have a bunch of metrics internally that we use, but we really wanted something that was industry-comparable, best-practice-proven. And as you know, there's many years of research from Google behind the DORA metrics.

So I don't want this to be a presentation about that. But for us, the big piece was how do we build these dashboards? And that was going in and getting all the telemetry out of our CI/CD pipeline, putting it into a data lake, and then being able to report against it with Power BI.

That turned out to be a lot harder than we expected. A lot of it's because of data integrity issues across the tools. What something's called in our pipeline, what it's called in the repo, and what it's called over here, they're not the same. And so that was a surprise, that we have a lot of messy data problems to deal with. So we're still working through that, as Bob knows.

But the dashboards are up for now for our four platforms we did in our pilot, and we can actually now measure their maturity and drill back in and see. It's funny, the maturity bounces around, which was not something I expected, but it does because some months are better than others. So they can be a level three one month and a level five the next month, right? I didn't expect that.

So that was really getting to that point. We are looking at some third-party tools for dashboarding because we really want to get more of the flow metrics, so that people can drill back and see where they're having problems. Maybe there's certain people on the development teams that need more training. Maybe there's a person who has a lot of defects. Being able to catch those things.

Because it all comes down to who's doing what, and how the work's getting done, and where are they struggling. We need to be able to pinpoint those. Because if you want a team to improve its maturity, you've got to be able to give that team the tool they need to figure out where they're getting stuck, what's causing their maturity level to be lower than expected.

So here's some of the questions we had as we thought about, as I came to this conference, what could I learn? What could I walk away with?

I wanted to learn more about automated governance, how people are shifting that left in their pipelines. We're a very regulated industry. I'm sure many of you are as well. We have a lot of things that just have to be done, a lot of compliance checks that have to be done. So that's a big one.

And of course, with AI coming in now, we're looking at, how do we protect intellectual property? Is there a way to automate that as well? Because we want to adapt and adopt the AI coding systems as much as anyone, but we're still worried about some of these intellectual property problems that are appearing.

Then how can AI also further benefit DevSecOps? Has anybody got any kind of wow examples of where they were able to use it to really see a big boost? Because if there's anything I can come back with and say, "Look, if we do this, we're going to get a big boost really quick because AI is making a big difference here." Maybe it's continuous testing. We've heard some of that with automated test case creation, and some of those, that could be it.

But I think there's going to be more done with AI in the pipelines. We're seeing all the vendors are starting to build it in now. Harness is building it in, ServiceNow is building it in, everyone else is. So we're looking for that.

And then finally, the one that's come up in a lot of conversations internally is innersourcing. This is sort of using open source, but inside the institutions, where you have intellectual-property-protected things that you want to build, and making reusable components, making them discoverable, making it so those AI coding systems can find them.

And then how can that benefit DevOps? Where can that benefit DevOps is what I'm really mostly interested in. I know how it benefits development, but how will it benefit DevOps?

So thank you for your time. We want to thank you for coming. We really appreciate you being here. Send us questions, or ask us, catch us later. We'd be glad to talk to you.