Las Vegas 2019

Staying the Course Amidst Change—An Update on Verizon’s Digital and Cultural Transformations

In this talk Verizon will focus on its state of transformation in 2019, including how it has changed its transformation efforts and started new ones to address these organizational changes, such as handling loss of tribal knowledge, re-earning the trust of our engineers, and improving culture through innersourcing and engineering recognition efforts.


Josh is a Senior Architect/Evangelist at Verizon, enabling development teams to build the cloud based solutions that make Verizon the leading communications company in the U.S. He does that by being at the leading edge of technologies, adopting startup mentalities and solutions, and applying them at massive enterprise scale to transform the way Verizon does IT. He also pushes community building and culture change within Verizon as MC and content coordinator for "Verizon Tech Days", internal conferences based on DevOpsDays.


Previously, Josh has been a software architect for multiple Verizon platforms used by thousands of developers including Cloud Foundry, Openstack, and Apigee. His passions are Cloud, Enterprise Architecture, DevOps, & API First. He has a B.S in Computer Engineering from Ohio Northern University.


Jacki is an Associate Director at Verizon, enabling development teams to build the solutions that make Verizon the leading communications company in the U.S. She does that by leading enterprise-wide initiatives aimed at creating a culture that passionately embraces modern engineering and organizational practices. These initiatives have resulted in the establishment of Verizon's Dojos and Verizon Tech Days (our internal DevOps Days); democratization of Verizon's technology decision process; creation of IT communities; initialization of an innersource program; and, adoption of Product Management practices and organization structure. Jacki has also created and led Project Athena, an effort aimed to increase diversity within technology at Verizon by creating a new labor pool sourced from underserved and underutilized communities.


Previously, Jacki held roles as a business analyst and project manager at Accenture, AllianceBernstein & Goldman Sachs. Her passions include Product Management, Enabling Cultural Change at Scale, & Diversity and Inclusion. She earned her B.A. in Economics from Bucknell University.

JD

Jaclyn Damiano

Associate Director, Verizon

JS

Josh Stone

Senior Architect/Evangelist, Verizon

Transcript

00:00:02

Up next is Jackie Damiano, who is associate director at Verizon and Josh stone is her colleague senior architect. There has been much written about the big changes that Verizon has. They make a massive bet on 5g investment. They will talk about how this has affected their dev ops transformation. That was years in the making. Many of us who have been through transformations that we didn't have necessarily have control over. This is a story of how they continued their mission, that they truly believed in, even in the face of big changes. Ladies and gentlemen, please welcome Jackie and Josh

00:00:47

Thank you, Jean. All right. Thanks everybody. It's really good to be back here at the DevOps enterprise summit. Again this year, I'm Josh stone joined again by Jackie Damiano. Um, like Jean said, I'm an architect in our cloud engineering organization. I also, um, was a founding organizer of dev ops days, Tampa bay. And I want to give a little shout out to dev ops days, Tampa bay, if you're in the Florida area and you want to know about dev ops, we want to be a part of that community. We're doing it again next year to look forward at dev ops stays Tampa bay.

00:01:22

So we worked for Verizon, a global telecommunications provider, and one of the major wireless and broadband providers in the U S to sum it up. We'd like to put it this way. Verizon delivers the promise of a digital world a little bit more about us. We're fortunate ranked number 19 with 131 billion in revenue and over 135,000 employees. Now we're well known in the U S for our 4g LTE network, leading the nation in coverage and reliability. And we're actively now building out our 5g network, building the first company in the world to launch commercial 5g service. And that's something that's going to continue to grow with our goal being 30 cities by the end of the year. And we're really, really excited about what 5g is bringing to us. It's not just another G it's not an incremental speed boost over 4g, and it is sure as hell, not just changing the icon in the corner of your phone. We believe the 5g is a driver of the fourth industrial revolution, the revolution of IOT, AI, VR, robotics, smart cities, and that revolution will change the definition of what a telecom is and does in the years to come. Here's just the bit of the taste of our 5g revolution. Let's roll the video.

00:03:49

Everything is about to change sounds ominous. What does that really mean? Well, it means that Verizon too has to change to best take on the 5g revolution. We've changed our org structure, our funding, our priorities, and our working model. This is such a massive change to everything about what Verizon is and does we call it Verizon to Datto and to Verizon, to Datto at its core is unifying and strengthening one of our greatest assets, our network unifying, and strengthen our network, and then transforming on top of it, the products and services we offer to our different segments, consumer business, and media media, a combination now of former Yahoo, AOL and other properties. Now, let me pause here for a second and help set some context here on where Jackie and I are. We work in the it portion of network and technology in our global technology services, strategy and planning, and then down in our cloud and engineering, cloud engineering and enablement organization, that's us.

00:04:59

Jackie runs our transformation office. I spend my days in the cloud, all right, back to two dot O in order to best position ourselves for success. In this revolution, we made some major changes to our working model changes that we wouldn't hadn't foreseen at the beginning of our transformation journey to become more nimble and flexible. In our engineering efforts, we partnered with a third party to take on a significant chunk of our ongoing development and maintenance of some of our systems, so that we could shift our investment into our future in 5g. And these, this isn't an overall leaner and more efficient organization. Now, another major change came with the organizational structure changes in the form of a voluntary separation offer. 10,000 Verizon employees took a voluntary separation, uh, offer over the past 12 months. And there was a significant amount of that within it.

00:05:59

And again, done to better position Verizon for investment in growth in our 5g future. But so while these changes are overall putting into Verizon in a much more competitive and advantageous position, they're changes that we hadn't had in mind around our transformation journey. And we started that transformation journey. So what happens when you're headed along in this transformation journey and all of these things start to change around you, and that's what we really want to talk about today. Three years ago, we started in earnest in our trends or digital and cultural transformation journey. And we've been sharing it here at the DevOps enterprise summit for the past three years as well. And each time we frame it into four areas, enabling technology, excellence, scaling patterns, and practices, fostering an awesome engineering culture and transforming our operating model. And each of these transformation areas was impacted significantly by the changes to our overall Verizon organization over the past year.

00:07:00

So what we really want to talk about with you here today is an update in each of those areas for Verizon, how it's been impacted and how we've adjusted and pivoted or push through in each to stay the course in our transformation in 2019. Now I'm the lucky one here. I've got it actually pretty easy. I'm going to give you an update on the technology. Jackie's, I'm going to take it and talk about our patterns and practices, our engineering culture and transforming our operating model. I've got the easy conversation. So enabling technology excellence in this pillar, really, like I said, I've got the easy part. This pillars we've seen the least change as result of Verizon 2.0 and the change, and the other changes here, we're keeping the same direction as always. We're just working to keep those external changes from slowing down the effort.

00:07:52

So a little bit about that effort about that momentum. We started back in 2016 by consolidating a fragmented ecosystem of SDLC tools, developing common platforms and putting the stake in the ground saying we are going public cloud first, along the way we grew those efforts, we migrated to cloud. We launched our dojo's. Um, we did a big bulk move of applications last year, and that brings us up to a point of now we've solved for the volume of bulk migration into cloud. We've reached that critical mass of workloads up there, but now it's really time to mature and modernize in that space. And when I talk about maturity, I'm talking really about three things. We focused on this past year, managing our costs, modernizing our architectures and continuing to build trust in public cloud. The challenge with that is doing all these things while still keeping up that momentum.

00:08:52

So let's talk about those costs. Last year, we said we were focused on that bulk migration of hundreds of applications, and the effort was really all around quantity. Just get the workload up there. Yeah, we got to pay attention to how much it costs too. So now that we've honed and strengthened that migration muscle we're really turned our attention to cost. In 2019, we stood up a centralized cost governance and enablement team, and that's created reports and provide a remediation steps and provided data that they then feed out to cost custodians. SMEEs distributed across the organization who actually truly take ownership of their cloud spend budget. So centrally we provide the data and the view and the incentive, but it's driven and owned in a distributed fashion. That's the only way we can really do this at scale and our big five areas that we focus on and managing costs.

00:09:44

And the levers that we pull are using auto scaling groups, terminating unused resources and terminating the tag-a-long resources, the unattached, IE, the unattached volumes. When you terminate that first, then you gotta get rid of that volume two, and then looking at the performance profile of each of these servers and saying, is it the right size? Can we shrink it? Can we change the format? Can we save some money that way and starting out with actually an undersized instance and then have application teams prove that they need to go with a larger instance to make a demonstrable. And then something just as simple as turning off unused resources, turn off the lights. We had a desktop virtualization environment and we realized, well, it's desktop virtualization. Nobody's working on the weekends. We can turn that off, save thousands. Now, light cost. Now that these applications have migrated, it's time to look at how best to leverage the public cloud.

00:10:42

That they're in what we brought to cloud, where mandalas in a fragmented ecosystem of different frameworks in middleware deployed manually via large, large release trains. So what we've done in our enterprise architecture organization is defined a north star reference architecture, which has guidance for teams to undertake and the undertake this refactoring from monolith to microservices, leveraging standardized frameworks, defined by the north star architecture with a preference to open source tools and using continuous delivery for that pipeline up to production. And this is something that's very early on. The north star architecture is defined. We're just now beginning to push the applications and teams and systems to adhering to that north star architecture. So lots still more to come in that area that brings us to our last technology focus area, continuing to build trust in public cloud with all of the changes that were happening.

00:11:45

It's very easy to want to stay still. There's a tendency to risk aversion. And one of those ways overcome risk aversion is to make a splash, do something, demonstrate performance and availability in a big way. So that's what we did this year, who here has the latest iPhone. Alright, excellent, good amount of people. They're great devices. And they come out. They come highly anticipated every year. Now Verizon's goal when pre-orders launch and the iPhone launches is quite simple. If you want to buy an iPhone from Verizon, we'll sell you an iPhone simple, but that's the availability and the performance of our website and our retail systems in this case. So we were taking our most critical customer facing retail systems during the most critical period of performance and availability, and now running it on the cloud. That's how you demonstrate performance and build trust in public cloud.

00:12:57

Let me show you a little bit of what I'm actually talking about here with this critical period. So they're at the bottom, that's a typical day of web traffic for Verizon wireless.com. This is the traffic spike that comes in. When I foam pre-orders launch, we plan the entire year to handle that spike. It is our peak moment of preparation and operations performance. So we ran it in the cloud dozens of applications this year that provide.com, our Verizon wireless.com, ordering retail device information, whole bunch more all ran this year for the first time in cloud. And we handled that spike successfully and we built, we continue to build that trust in public cloud. And it's really been an inflection point with the discussions that we've had with teams both pre and post this pre-order in. Oh yeah, the clouds, the real deal. It performed at handled the task in this kind of spike.

00:14:04

This is exactly what cloud has made for, right. We S we beefed up the resources a few days in advance. We ran through the spike, we handled it. And then we turned them off for a fraction of the cost of what it would be to acquire and host that same monitor of hardware in a data center running year round. Just for that, that brief, brief window of that spike. So that's what we've done this past year. Where are we going from here? Well, to best support, enabling modern application architectures, we need to ensure our platform offerings. And so what we need to do now is we, oh, we offer Kubernetes. What we need to do now is mature. And like we did this year mature in cloud. We need to mature our platform in Kubernetes as well. And in order to do that and to really drive and participate in the cloud native space, we're proud to announce that Verizon is now a member of the cloud native computing foundation, making contributions to Harbor and the part of this, uh, application delivery, special interest group. So that about wraps it up for technology. So I'm going to hand it over to Jackie to talk scale culture and operating model.

00:15:12

Thanks, Josh. Okay. So Josh was right. I do have the more difficult part of the presentation, not solely because I'm in three inch heels, but also because the things that I'm going to talk to you about today mainly deal with people. So I'm going to cover the remaining three pillars of our digital digital transformation at Verizon. All right. The way we scale patterns in practice at Verizon is through our dojo's. I'm sure that all of you have attended a session this week on dojo's. We talk a lot about them. We've seen a lot of success come from dojo's three years ago, we instantiated our dojo's at Verizon. The literal translation of dojo is the place of the way we have two dojo's, one in Florida, and one in New Jersey engineers go there to really get to be masterful at their craft. They are permanent locations in each of those states.

00:16:17

So at Verizon, our traditional dojo models look like this teams come to the dojo physically for usually a six week engagement. We provide them with some really amazing coaches who help them upskill. They teach them practices like TDD, et cetera. The calling card of the dojo though, is that everyone gets together in one physical location, and they actually learn how to these new principles and methods actually within the context of their work. So it's not like they're building Lego's to try to learn TDD. They're actually in the repos and the coaches are sitting alongside them, helping them understand those practices. So usually they come in six weeks, we bring them in to pizza, full stack teams. This model worked pretty well for us for the, for the first couple of years of dojo's. But with all of the changes that Josh talked about, we started to face some significant demand issues.

00:17:13

This year, this traditional model at Verizon assumed two things. One is that we could physically co-located the teams two is that the teams could get their heads around the fact that sometimes you need to slow down in order to speed up. Now in this new world where we have a lot of people turning over, we have great amount of turn in our staff. We literally lose thousands of colleagues over a six month period. The amount of knowledge transfer that has to happen during that was just two great teams. Truly didn't have the time to slow down to speed up. Also, what we saw is that now more than ever our, our teams became even more distributed. So we had a pivot and we had to do it fast.

00:18:09

All right. So today we go to them, we're meeting our clients where they are. We are embedding our coaches in the it business units. It allows them to increase their competency of the tech stack for that individual organization. It allows them to build the rapport with the teams that they need in order to provide them with advice that will make them better. It's helping us shift from a know-it-all culture to a learn it all culture. We also are not drawing a hard line about what types of coaching we're doing in terms of number. We can be doing one-on-one coaching. We can also do team-wide coaching. So this is what we've shifted to and looking at it, it looks super obvious. Now, nine months ago, not so much, we're trying to figure out how to, how to make this pivot and what to do. So it provided us an opportunity to change how we were using the dojo.

00:19:09

So I'm going to pause here and I'm going to give a little bit of context a year ago, I was on this same stage doing an ignite talk. I talked about my concerns about the lack of visibility for women and people of color in tech. It's because of this conference that my life and my career have actually pivoted in the past year. That may seem hyperbolic to some of you, but I can assure you. It's true. Those of you who came up to me and talked with me after that talk and shared your own stories, you really helped me create this burning platform. Once I left here to go back to Verizon and to do something about what most people just talk about. So now I'm going to introduce you to something that we did with our dojo's. That was pretty radical meet project Athena.

00:20:01

Now I realize if you've had a drink with me this week, or if you've seen me in the bathroom or in the hallways, you've probably gotten an hour from me on project Athena. Um, but for the rest of you, I'm going to go ahead and tell you what this project is all about. Quite simply, we are not diverse enough in tech. We talk a lot about it. We don't do a lot about it. We need to make true investment and that's how change is going to come. So what we did at Verizon is we decided to go into underserved communities where our dojo's, our partner with community organizations like unemployment offices and the salvation army, identify people from these communities that underlying characteristics and skill sets that we believe will make them great technologists. That didn't mean that they had to have their PhD in comp side from MIT.

00:20:58

We were actually looking to create a new pipeline. Listen, out of all of the people that graduate with comp side degrees, 18% of them are women. 18%. We're all going after the same 18% of women and the numbers don't get much better when it comes to people of color. So we went out, we partnered with these community agencies. We identified people over 400 people applied for 40 positions at Verizon. I'm super proud of what we've done at our organization. We got sign-off to co locate these 40 people, 20 in each dojo. We brought them in on June 3rd. They began, we brought them in as contractors key. We paid them a living wage to learn a lot of these people couldn't afford to stop working at their survival jobs in order to create a career, not just temporary wealth.

00:22:00

So we co-located at them. We put them through an online full-stack JavaScript program. That material was augmented substantially by our in-person coaches. There were four people who are amazing in every way possible. And I loved each of them before we began this journey. And afterward they are more like my family. So Steve Thomas is one of those folks. I know a lot of you in the community know him, Nate Ashford, who you heard from last night during the ignites is another. So after we co located them, after we train them up, June through October, we graduated them with their full stack, JavaScript degree or certification from the program. And then we embedded them as apprentices inside of our tech teams out in the lines of business. So today, right now our apprentices are working in Verizon code basis. It's funny. I had a question that said, uh, Jackie, you know, it was four months enough, are they actually able to, to contribute at this point? And I said, well, two weeks ago, one of them did push code to prod. We have another two that created bots. So we asked, yes, it's working. They do not need a PhD in engineering in order to learn and to contribute to our technical teams.

00:23:31

So you can see now why I said that it may have sounded hyperbolic, that I said that my life and career has changed in the past year, but it's actually true. So we brought 40 people into our, we expected a 50% attrition rate through these six months, we have gotten a 10% attrition rate. We were really wrong. Um, and what we're doing now is that at the end of November, we're posting positions. And it's our belief that the vast majority of these apprentices will interview successfully and be converted to full-time employees by the end of this year, fundamentally changing their lives and the lives of their children. Okay, thanks.

00:24:19

So here's one of our cohorts. This is our Irving Texas group. I once heard that diversity is a diverse topic. It really is. So we have a great mix in terms of gender, race, ethnicity, interestingly, religious beliefs, our apprentices range from 20 to 63. And very important to me, to me, socioeconomically, we have a very diverse group. This group of folks, it's the hardest working group that you'd ever meet. Um, a lot of our Texas team actually came out of our relationship with the salvation army. So I will tell you that these folks deal with Maslow's hierarchy of needs issues. We had people who were struggling with housing, with food, with getting to work. We had one woman who was on a bus three hours each way to the office. And she now is sitting in a role happy as can be contributing. And her manager couldn't be happier about, you know, what she's doing now through the program and through what Verizon has done and funded, she's been able to get a car.

00:25:33

I mean, it sounds so simple, but geez, it's huge. So there were a lot of circumstances that actually happened in order for this to become a reality. One of them is that I'm extremely annoying or I guess persistent, depending on your perspective, when I really believe in something and I really, really, really believe in equality and diversity. Now, our dojo's had little demand as the manager of that team. I needed to figure that out. Verizon also had to hire a lot of people this year to backfill some of those people who chose to take the voluntary separation opportunity. We had a hire a lot of people in a short amount of time. And what's interesting is if you're one of the companies who's growing really, really well or fast right now, you may not realize it, but you have a really important, um, environment that you're dealing with. You have the T the, the chance right now to get the math, right? In terms of diversity, if you're hiring a ton of people, get it right now, it's harder to fix five years out. So the other thing that really helped me sell this was that Verizon just declared social responsibility, a core value. And honestly, again, I'm super proud of that. I want to work for a company that believes that we owe something to society, to our customers.

00:26:58

Lastly, I said it in the beginning, a lot of people talk about this. Verizon was willing to give us the money to actually fund this program. They're walking the talk. Okay. All right. Now I'm going to go on to our engineering culture pillar. Now I told you that there are going to be a lot of highs and lows, and it's going to continue. So here for the past three years, we've made significant investment in making sure that our culture at Verizon was empowered, engaged, and excited. We did things like have internal dev ops days. We call them tech days. We sponsored external, uh, functions. I can conferences to make sure that we're showing up at the right places. We encouraged our people to speak at events like this one, then Verizon to Datto happened. How many of you know, someone who has left a company that you really sort of felt that loss to show of hands?

00:28:01

Okay. Now imagine that that feeling is amplified, not by just losing one colleague, but by losing 50 a hundred, a thousand colleagues in six months, a lot of us lost people that we felt were more family than colleagues. And a lot of us, including myself, lost my entire sponsor network over a span of three months. It was a really difficult situation to be in, but there's light. So listen, we doubled down on the things that we did well. We reinvested in our tech days, three times a year. Now we're going out to our technical hubs and we're getting people excited about the work that they're doing. We're sharing with them, the good stories and the stories that we've learned from. We bring external luminaries in to talk to our engineers about the latest trends in tech. And then importantly, we ask our more junior people to get up on the stage and talk about their experiences, the good, the bad, and the ugly.

00:29:08

What's also super cool is in the past year, we've also seen a lot of new blood come in and they have a lot of really great ideas. It's helping us with our culture. We also brought the Verizon media folks closer to us. So these are our people from AOL oath. And that's really helped because a lot of what we're trying to do with our culture in terms of encouraging experimentation is something that they did. Um, as a digital native, you could see here, we've had some impressive stats, three years of running and people love it. They look forward to these days. All right, the next thing that we double down on inner sourcing, if you're not familiar with what inner sourcing is, it's basically open source within a company. And our sourcing gives you three things. One is it reduces duplication of work. That same address, validation that all of you are trying to create across the organization. We don't have time to be wasteful anymore, and to have people duplicate that. So inner sourcing allows us to share those, those functions. The next thing that it does is it decreases time to delivery now in get lab, which is our tool that we use. You can actually go in and create the change that you need and propose a merge request that then takes the dependency that you have off of that other team. So now more than ever, we need this inner sourcing culture because we're lean.

00:30:36

The last thing that we're doubling down on and focusing on is transforming our operating model. So listen, if you only focus on tools on a digital transformation, you're not going to see the benefits that you would, if you also concentrated on how those tools are used and how your engineers are adopting those tools at Verizon, we're experimenting with the product model. You've all heard from Mick Kirsten and others who have done a lot of really great work in this space. We're trying to move optimization from delivering on time and on a schedule to making sure that you're delivering the right thing. And that the is at the center of all of your work. Right now, we've narrowed our experiment down to our internal platforms teams. So we have five product teams set up. The journey has not been easy with all of the staffing changes, but we fundamentally believe that it will accelerate time to value.

00:31:36

So Jonathan smart made fun of maturity models last night. And so last minute we had to change the title of this freaking slide. Thanks, John. Um, how are your teams doing? This is not a maturity model. No, actually. So it was really important to us to actually put something like this together. Um, a guy on my team, Brian Mae is amazing, super smart. Um, he went through and he created this. We needed to give our people sort of a north star picture of what this looks like and how good you could get. And so this has been helpful when we're having that conversation with our internal teams to really help them orient around where, you know, where they are on this journey. We're looking to open source this. And also there's another piece of code that I neglected to mention on the inner sourcing piece, but Connie Rogers built something that was super cool and helped us actually open our repos at Verizon, which is a data leakage, uh, protection utility, um, which scope scans code for any secrets that may have been there. So those are the two things that we're going to open source, and you'll be able to get ahold of them after the conference.

00:32:46

And I skipped a slide. All right. So we can all take a deep collective breath. My rollercoaster is done. Thank you for going on the journey with me. Um, look, the past year has been really tumultuous in good ways and bad ways. I think that the obvious question that people have now is, Hey, you know, what made you stick in there? You know, what, what kept you going to work? And I look around and really it's all of these bright lights, right? And when I think about Athena, that project is probably the most important thing that I've done in my life beyond choosing my husband and having my children. I've been able to work with 36 hardworking people who just needed a door that they could walk through in order to change their lives. So if I keep getting opportunities that I think are fun and interesting and challenging, and I work for a company that lets me do that I'm in every day, all day long. So that said, we have a couple of things we need some help with. I have now been zeroed out twice. Jean I'm probably gonna end up on some naughty list, but here they are hit Josh, Josh, and I up, if you have any questions, we're more than happy to help, um, and have fun trick or treating tomorrow. Everybody. Thank you. Thank you.