00:00:07

For years, I've mentioned that one of our goals within the DevOps enterprise community is to have business leaders co-presenting with their technology leadership counterparts at this conference to validate that the work of this community matters to people who matter as my leave-in from American airlines said earlier this morning, this year has been a whole load of 2020, but without doubt, one of the high points for me was meeting the next speaker. And I was so grateful when she told me she was willing to present here. Kimberly Johnson is executive vice president and chief operating officer for Fannie Mae, which is in the fortune 25 list of companies. Their mission includes making home ownership and affordable housing available for all Americans for decades. She has earned the reputation of being asked to solve the toughest problems facing the organization. And I think the story of how she ended up being responsible for the technology function is profound, which is changing how technology is integrated into business strategy and operations. I can't overstate how much I've learned from her in every interaction I've had with her. And I've asked her to help teach this community of what we need to know in order to help our organizations win. She'll be co-presenting with Chris Porter, SVP and chief information security officer, Tim judge, VP of climate impact and Ramon Richards, SVP of integrated technology solutions. So here's Kimberly Johnson and team

00:01:38

Welcome. I am really excited to join you today here at the DevOps enterprise summit. We're looking forward to providing some new insights, challenging ourselves, to think differently, making some connections, and of course, sharing some stories about the journeys that we've all been on along the way. So I looked through the list of attendees and there are some really amazing companies here. Um, I'm sure as you looked through the list, you were wondering why is Fannie Mae at a DevOps summit? And some of you were probably even asking, well, what is Fannie Mae anyway? Well, I'm going to spend a little time this morning talking about the role of Fannie may have dev ops has helped us advance some solutions in the face of rapidly changing environment, but we're also going to talk about the Fannie Mae's version of dev ops, which is really DevSecOps and how we incorporate security into everything we do.

00:02:24

I'm going to pull in some of my female colleagues along the way to help tell these stories. And we're really going to drive home the theme of that connecting business and technology. So first, what is Fanny Medi? Well, we like to describe ourselves as a secondary mortgage market company, but those words don't mean a lot to most people. The gist of it is banks lend money to people so they can buy homes. And we buy those, those loans from those banks so that they can have more money to lend more people to buy more homes. It provides liquidity to the entire mortgage market. What we, with all those ones, we buy, package them up, put them into securities and we sell them into the capital markets. And this is great for homeowners. It means that it can be more affordable. It makes it cheaper for them to borrow instead of borrowing money at the rate that any person might be able to access for the risk that you might get for buying home.

00:03:21

You take thousands of borrowers, you package them together. We put a guarantee on those cash flows, and that gives a really reliable security that investors can buy. And they put billions and trillions into the market to help support housing for everyone in America. And everyone in America is a really important theme for us. We believe in helping all more, gets all times Fannie Mae and its size and scale is something to really wonder. We have a three and a half trillion dollar balance sheet and housing is up to 15, sometimes 18% of our annual GDP. So the bottom line is housing is important to the economy in America. Now this is great in terms of scale, when it comes to us, delivering customers solutions, but it's tough when you think about change, changing something that's so deeply embedded in the economy of the country, it means you have to be very careful.

00:04:15

It's created a risk tolerance for us that doesn't necessarily help drive innovation, speed, adaptability. The things we're looking to get out of a fast moving experience. When we started looking at the way that customers interact with us and the experience that they get and getting a mortgage, we realized we really have to start evolving and they have to start evolving. Fast customer experience is really changing. They're expecting to be able to buy things from their phones, with their fingertips in, in moments. And we still we're working through a process that was full of paper and forms and signatures and faxes by faxes. So we've been spending the last part of our digital transformation, trying to digitize the front end, but also try to digitize the entire process, making things faster, easier, simpler for the lenders and for the investors who are part of our mortgage ecosystem.

00:05:06

So before I launched into our DevOps story, I want to tell you a little bit about myself and my journey at Fannie Mae. So I had gene Kim earlier this year, come and talk to my entire company about dev ops. And as he began to learn more about Fannie Mae and how we've come to value, sort of speed, agility, locality, simplicity, flow, the things that are really core. He asked me a really important question. He said, Kimberly, I love that you're involved with this technology, but you've told me your story. You've been here at Fannie Mae for 14 years. You've done it, you know, capital markets, multifamily, and single family and risk. And why are you the COO? I don't understand how you're leading the technology team. And I have to say, I took that as a compliment. They're like, what makes you qualified to do this job?

00:05:51

The question can come off a little prickly, but you meant it in the nicest of ways. And I had a really ready answer for us. It's not about having the best ability to code or knowing how to provision servers. The leadership of our technology team is about bringing together business and tech. We recognize that this moment that it's not going to be about us and them, there's only going to be one business in the future, and it will all be powered through technology, fusing those things together. It's going to be what makes us compete and win in the future. I have to say when, when I started in my role, we had a long history between the business and the technology teams. The trust was pretty low, and I would say the empathy was pretty low too. We had a history of taking too long, costing too much and not quite getting the mark when it came to delivery.

00:06:41

And so there was, you know, blame on both sides of that equation, but it wasn't a harmonious co-location of people working together towards the same goals. Number one thing we did when we started off on our dev ops journey was make sure that we were actually connecting our customers, our business people and our technologists so that we could all be going after the same thing. I brought my entire management team along with me. I mean, we really went all the way to the experts. I had people like George Westerman and Jeannie Ross from MIT coming in and talking to our board of directors about what it means to be digital and why. Um, we had them doing classes with us at MIT, talking about how to bring digital into the world and the difference between digitizing and a digital transformation. And lastly, of course, we had the illustrious team, Kim, we brought him in to talk to our whole company, not just our tech teams.

00:07:32

We wanted everyone to understand what it meant to be on a dev ops journey and believe it or not, it was Brivity. He was explaining to people of all levels and all businesses, why we needed to tackle technical debt. And they were just wrapped listening, understanding and getting to the point. So this has been something for us that has been not just a technology journey, but a company-wide journey. And that's why it's starting to stick. So we've now aligned our devs ops or dev ops journey and our cloud journey together. And as we're putting things in cloud, we're shifting our teams with more DevOps tools and practices so that we can actually become that future, that we had that vision of when we started this a year ago, another funny thing that you want to focus on, once you get your alignment, you know where you're going and you get all your foundational pieces, right.

00:08:22

And you have everything set, get everyone on board. And the first question becomes, okay, great. Is it working? How about now? Is it working? Is it working yet? Are we there yet? Are we done? I found that as soon as we got ourselves launched, we got funded, we got moving, we got people actually aligned hands on keyboard coding. We got our CIC pipeline in place. We put our, our, um, microservices architecture. We had our streaming data platform off to the races and not three months later from the top of the house and the board to the management committee, everybody said, how about now? Is it working? How do you know what's working? Where's it going to show up? How will we know? And that is a really good question. How will we know for us? It's about metrics and metrics have been a journey, too.

00:09:07

We started off trying to measure a handful of things. And what you find is you get what you measure, but that means you have to set your measurements, right? And that's been a continual evolution, a process of continuous improvement on measuring the right things to get the right results. And so I would say that, um, you know, you gotta put in a lot of elbow grease to get the right metrics for us. It has been a learning experience. We focus on things like lead time and meantime to restore. And even on, on the software failure, deployment rates, some of the key DevOps metrics, and those have been so volatile, you're going to change something and you get it just right, but it doesn't change the behemoth of the company and the way you measure everything, it only changes in one place. So how do you get your metrics to start moving in the right direction?

00:09:50

And people can see the progress that you're making. So we decided, oh, well, we'll split out the metrics, the old world, the new world, and we'll compare. And that'll help people understand the value of what we're doing with dev ops, but we can see the difference between what we measure and the new world and the old. And then I'd say, lastly, we had to do a lot of work, breaking down our metrics and the things that people could understand. We said, you know what, every metric isn't the same there's metrics that measure inputs, are we doing the right things? Do we have the right guideposts? Are we following the right practices? We had measured around execution. Are we, are we living the plan? Are we doing what we say we're going to do? And then we had measured around outcomes. Is it working? And we helped everybody understand.

00:10:30

You can measure your input metrics really quickly. You can measure your execution metrics along the way, but those output metrics take a while to move. And that started to help. And we stopped having the answer. Are we there yet? Are we there yet? Are we there yet? All right. I guess last thing I would say is just about communication. Oh my goodness. Everybody says it takes 10 times. 10 times is a very low under estimate of how long it takes for things to really sink in. Especially when you're talking about changing this magnitude for us, we anchored on agility. We want to be able to do that quick responsiveness. We don't want to fall into the same traps we did before. Um, or I'm not being able to, to integrate new risk management techniques. We, we want to be able to move fast and agility has been the underpinning for us.

00:11:17

And it's been a year of telling everybody every time anybody asks, why are we doing this again? It's for the agility. We found that if we focus on both agility and efficiency, you can get to some of those results that you were CFOs, really like for us, focusing on, uh, retirements, turning things off fastest way to save money, turning things off fastest way to spend money, build everything twice. So there's a really important trade-off about creating the new, turning off the old and very last, I would say scale, we spent a year getting our dev ops journey, right? And we can finally see some of these things coming together, but you find that getting it right. One place is wonderful victory, but getting it right everywhere is a hugely daunting task. And so we're on the journey too, of figuring out how to take those small nuggets of really great productivity that we've developed by putting our DevOps pipelines in place and having our teams smaller and autonomous autonomously designed so that they can actually deliver on a more continuous basis has been terrific for morale terrific for productivity, terrific for results, but getting that from one or two teams to hundreds and hundreds of teams.

00:12:29

None of that's the question that we really want to answer. And, uh, I'm excited to have some of my friends tell you about how we're doing it. So with that, I'm going to bring on Ramon Richards. He heads up our development teams. He leads our integrated technology solutions group. Rowan, my first question for you, can you please share how that COVID-19 reaction and everything we had to do to bring for Berets to borrowers has translated into technology and operational challenges for us and how DevOps helped us address those challenges.

00:12:56

So the impact of the pandemic required Fannie Mae to quickly figure out how do we deliver new solutions for our customers in a matter of weeks to ensure we provided the help they needed, um, before they were adversely impacted by the crisis. And so while we're used to delivering for customers, the speed in which we had to turn around to respond to this crisis was new for us, but our confidence was high and our confidence was high because we had made an investment in dev ops in our servicing area. And, you know, we, we were confident we could deliver the solution, but there were a number of challenges we had to immediately respond to. The first challenge was finding the capacity to take on this body of work. We had a number of high priority items we were already focused on. So we were able to identify the teams that we would reallocate to focus on.

00:13:49

Um, this high priority work. One of the challenges in the past when you reallocate teams, is that you have development work that they already have in flight that you have to put on hold. But because of our commitment to the dev ops practices, we have been delivering production releases on a regular basis. We have a continuous delivery process. So it minimize the work that we had to put in on hold, and it allowed us to reallocate teams, but still deliver value to our business partners. The second challenge was we were still maturing along the DevOps, uh, curve. So we hadn't fully arrived yet, but we knew when we needed these teams to quickly be high-performing. So what we did was we, you know, assess the talent that we had on the teams. We were able to, um, swap in some more seasoned dev op experts where we needed to. And we also partnered our teams with some coaches who were very seasoned in the DevOps space as well, that helped us accelerate our maturity in a short period of time and positioned us to deliver the capabilities we needed, um, in a matter of weeks.

00:14:58

Ramon, thank you for that. That makes a lot of sense. Does dev ops help us more broadly in servicing loans beyond just what we're dealing with in COVID-19?

00:15:06

So there are certainly benefits beyond just the crisis that we were responding to. So the adoption of dev ops has allowed us to think and operate differently with all of our partners within the servicing space. So one of the important concepts is integrating our product management teams, our development resources, our site, reliability engineers, as well as our partners like architecture and information security, where we're all at the table together, which reduces handoffs, reduces communication and allows us to make decisions in a timely manner. Um, it's changed the way we think about our testing. We shift a lot of our testing earlier in the process. We drive automation as we are developing our solution, which speeds up our ability to deliver a quality product. Um, and then we have a continuous, uh, bill process. So on a daily basis, we are building the software, um, that we are creating to ensure that it is high quality, um, and that we are learning as we go and we're adopt adapting pretty quickly. So a of those benefits played out as we executed to turn around a solution quickly, but it's also benefiting, um, other work that were other important work that we're doing in the servicing space and really driving us to be more agile and delivering, um, with a faster time to market.

00:16:30

Excellent, well, fast cycles and the quick learning loops. That sounds really good. Well, if you had to summarize the big takeaways for us, when we think about COVID and dev ops, what would you say would, would be the things you want people to know?

00:16:44

I think one of the things that's really important as that, uh, you know, the team that is focused on your, uh, delivery and our important stakeholders and how you get the job done that, um, they all are operating with the same mindset and they all understand, um, the principles and the processes that are part of the DevOps process, that it is about, um, collaboration, early engagement in the process. It's about automating everything from start to finish. Um, it is about, you know, timely decision-making. Um, and so, you know, we, we have found through the experiences we've had, how effective, um, how effective dev ops has been in, uh, ordering, allowing us to deliver. And the next challenge for us now is to continue to scale these practices, continuing to build, um, that, understanding the mindset, um, the understanding of how you leverage the tools, um, across the,

00:17:50

Thank you, your mom. That's terrific. All right. So moving into our next segment, we just finished up how females use dev ops to be able to have more agility in managing some near term things that have emerged like COVID, but we've also been looking at things that emerge over a longer time horizon. You remember Fannie Mae makes loans 30 year loans. We are, uh, all of our loans are backed by collateral houses. And so as we see more weather events and climate change, there's a lot more risk of things like wildfires, hurricanes thing, that damage homeowners, things that have risks for loans that last 30 years that are backed by properties. So with that, I'm really excited today to introduce you to Tim judge, he's our top officer for climate risk. And we're going to spend a little bit of time today talking about how DevOps has also helped us with some of our longer range goals. So, Tim, it seems like we see these types of new weather events happening more and more frequently. Now what's Phantom me doing to stand up capabilities to be able to handle this and how can we respond?

00:18:49

So the first thing I would say is the intensity and frequency of severe weather events are certainly increasing. And so when we look at the program, we knew we had to stand up a more holistic program around climate. Now, the first challenge there is very similar to digital transformation is how do you address day-to-day severe weather events while still building a longer term program at Fannie Mae? I was very lucky to have already a disaster response recovery team that Fannie Mae had already stood up that was able to take care of things on the ground when natural disasters occurred and help communities. So that really allowed me to focus more on that medium term and longer term build out. And then when you look at the longer-term, build-out, it's really about breaking that problem down. So when we took over the climate space, the first thing we did is say, where do we want to focus our time and our energy first to bring value.

00:19:43

And we found is you break down the portfolio, we have, like you said, three and a half trillion dollars of balance sheet, and that's about 18 million loans, right? So there, you can't tackle that all at once. So we broke it down and we said, what part of the portfolio was most at risk? And what risks are we really going to focus on? First, for instance, are we going to focus on flood versus wildfire versus Hale and severe storms? And then the most, the other important thing is we brought that plan to the board and made sure the board understood what was our short term delivery, where they would see quick value out of the, uh, the climate program, but then also make sure they understood the longer term, uh, transformation journey that we're going to have to be on. Because I think that was a really important part of the socialization is giving people quick value, but also making sure that they understand that it's going to be a significant journey.

00:20:39

Awesome. Thank you, Tim. That, that makes a lot of sense. And Fannie Mae has been at the heart of this digital transformation. We've been talking about this as we go through this conversation, how has that digital transformation and dev ops specifically helped you guys as you've been building out this team that's facing all of this a really interesting and evolving climate risk?

00:20:58

Yes, I'm a little bit different than most business folks in that I came recently from the COO. So I am dev ops is kind of in my blood about me wanting to be a sponsor for that. Uh, but I would say a couple of things about dev ops is for me tackling, like I said, a huge climate issue. One of the things that really helps me is the speed and agility to quickly experiment. The climate world is changing so fast. The modeling of these climate events is really evolving very quickly. So I have to have an environment that I can quickly turn and quickly look at different models. And dev ops gives me that the second part of it is there's a big regulatory part of climate and the expectations from regular regulators and investors on disclosure is only heightening that gets to the risk side, right?

00:21:48

I want to be able to get really good disclosures to the industry, which means I need to deliver high quality metrics on a regular basis. And then the other thing I think we've talked about a little bit before, also on dev ops is scale. I'm running lightly for the climate. I'm running those 18 million loans. I'm not running them once. And looking at them, I'm going to have to look at 18 million loans and what climate does to them over 30 years. So you can imagine the scale that that has to date in terms of data footprint and in terms of compute all those things aren't had possible. If we were still living in the old world of the basic infrastructure that we used to have, you know, dev ops gives us that ability to be really nimble. The other thing I think it really does right now, it's now it's almost a perfect time for dev ops because with COVID, there's a lot of, a lot of the industry's really challenged on budgets. We talked about the CFO wanting to see something well with dev ops, you get to walk into the CFO and say, Hey, here's some quick value that I can deliver. Here's a lower risk program. So you're putting your money in a lower risk area. And then finally, it's more responsive. So I don't have to tell you here's my three-year plan and stick to it. What I'm really telling you is here's my plan every single quarter, and you can know that I can respond as needed.

00:23:11

Excellent. Thanks, Tim. That was really insightful. So we've talked a little bit about COVID-19, which is sort of a near term, fast moving crisis. We talked a little bit about climate change and how we're tackling sort of longer-term more drawn out crisis. And as I think about how we're putting all this together, it reminds me that the one thing that underpins it all is security for sit for Fannie Mae safety and soundness is part of our mission. And we're a company that has gone through a crisis before. I think I, I mentioned at the beginning, we have a three plus trillion dollar balance sheet. We helped finance more than one of the four homes in America. We have a really important mission that we just can't put in jeopardy. So our risk tolerance has been somewhat low over the years and that makes all of this change really, really challenging. And it makes security very, very important. So with that, I'm asking our CSO Chris Porter to join us. And we're going to ask him some questions about how we're managing through this, uh, evolution, this transformation that we're making and how we're incorporating dev ops into everything we do in a safe and secure way. So, Chris, it sounds like infrastructure is a really important part of security. Can you tell us a little bit about how security plays into the role of test and learn in this environment?

00:24:23

Yeah, certainly, you know, I think about this in a couple of ways. Um, one is about culture and changing the way that secure security communicates with our development teams. Um, and the other is about how we integrate security tools. You know, when it comes to culture, we we've had to change quite a bit like ourselves. You know, the, the old way of security is that we did our own tests, you know, that was handed off to the security team. We ran our own tools against the apps, you know, we collected the results, the vulnerabilities, um, and then we handed these back to the dev teams and this made for a completely inefficient process. Nobody liked it. Uh, you know, by the time the code was sent to us, it was ready for production. Um, and now they were waiting on us to do our tests and we would take our sweet time, further delaying the delivery.

00:25:08

And then we would hand them a giant spreadsheet of vulnerabilities and said, good luck. We need you to fix all these things, you know, so we had to figure out, uh, you know, how to move left in this process. How do we, how do we shift left in the development process? And so we did this by relinquishing the control over our precious security tools, you know, make them much more self-service API based, integrating them into JIRA and Jenkins and allowing developers to run, run these themselves. Um, so it took a lot of time, you know, we had to train developers on how to use and run the tools we had to teach them how, what the results meant. We also had to change our nomenclature. We stopped talking about vulnerabilities. You know, we started calling them defects because ultimately security vulnerabilities are just another kind of, of code defect.

00:25:56

Um, and then we, we need to continuously get better and, and fully integrate all of our tool sets within the CICB pipeline. We need to integrate all the security tools, like a static analysis and dynamic analysis. Configuration management checks, compliance checks all needed to be in the pipeline. So that every time that code was checked in, we were running a test. Um, every time that, uh, we were doing a delivery, we were running a test and this makes it easier for developers to know what they need to do. Right. You run a test, it fails, oh, I have to fix something. Uh, you know, we call this, uh, you know, I call this like the paved road, you know, if you follow the paved road and you use the CICB pipeline, which has all the checks and integrated into that pipeline, then it's going to be much easier for you to deploy code.

00:26:43

And we treat it like an ad online, right? Like if, if the test doesn't pass, it breaks, the line has to go get fixed and then delivery can continue once it's fixed. So if you follow the pipeline, if you don't follow the pipeline, then it's, then it's the Rocky road, you know, it's slow, there's lots of potholes. You get pulled over by the locals. No one wants to go that route. Uh, and, and it'll take you much longer time to deliver value to the customer. Um, and a little note to, to, uh, gene Kim and the unicorn project, uh, it will, it takes you longer to get joy to the developer as well.

00:27:16

Excellent. Well, I think that's terrific. The idea of automating everything, especially those tests, that's so important and you gave us some really good tips there. I got one final question for you. It's about mindset from the CSO perspective, what mindset do you think developers need in order for all this to come together successfully?

00:27:35

No, I, I actually think there's a mindset change that has to be made by both, uh, the development teams and the security teams. And, uh, I saw this tweet recently, um, that I think really drives this home. It's from, uh, Richard Searson, who I think is with a company called soluble. And he said that DevSecOps is an observation, not a title. And it's, we don't care who does the work or where it sits in the organization. It's just how security is done in modern organizations. So on the security side there, there's, there's always been this mentality, you know, that you must protect the developers from themselves, but when you move to a dev ops model, you move to the, you build it, you own it, you know, those full stack developers operating there. Well, you know, what else is part of the full stack security security is part of the stack.

00:28:27

So there's gotta be this kind of shared accountability for security between the dev ops teams and the security teams. And then on the security side, you know, we have to do our part, uh, w w we've got to deputize those dev ops folks. We've got to help them. We've got to train them. We've got to build programs like security champions, where on every single dev ops team, there's somebody, at least one person that's accountable for making sure that security is implemented, you know, on the security side, we can't scale, right? There's, there's a supply demand issue in our industry for security folks. Uh, th th the demand is far outstripping the supply. So we, we've got to do our best to bake it in and then train the DevOps teams on how to own it and, and implement it the way it needs to be done to appropriately manage risk.

00:29:11

Awesome. Thank you for that, Chris. That was really terrific. So today would've had a full conversation around how dev ops can help us respond quickly to the evolving world around us here at Fannie Mae, we are preparing for today for tomorrow for 2050 years from now, we're rebuilding our infrastructure and a brand new way to meet all the challenges that we see ahead. And then we looked at some fast moving things like COVID some still removing things like climate, some really essential foundational things like security. And we shared a few of the stories you, so our big learning is that dev ops is about how we do everything. Business technology, completely intertwined. It's how we get it done. And the lesson that we've learned, isn't so much like you've got to design the way that you do things the right way. It's around the change management.

00:29:59

That's required to get everybody to adopt the new way of doing things. And for us change, management's been around communication, making sure that you have alignment of vision or metrics, making sure that you're executing as, as you expect, it's been around scale, getting it right once, and then being able to scale it up and around continuous learning so that you can get better and better. Nobody gets this right the first time. So, so those are the lessons we've learned on our dev ops journey. And I hope you've enjoyed what we've shared with you today. And we look forward to many more DevOps conversations in the future.