Las Vegas 2018

Transformation at Scale

Driven by a period of significant disruption in the telecommunications industry, Verizon is in the middle of a massive multi-focus transformation effort to become leaner and more agile in it's application of technology. In this session, Jaclyn and Josh of Verizon’s Global Technology Services organization will detail the current progress of this transformation in the areas of public cloud adoption, technology modernization, and culture transformation while sharing the unique challenges and successes of driving such a transformation at the scale of Verizon.


Jacki is a People Leader at Verizon, enabling development teams to build the solutions that make Verizon the leading communications company in the U.S. She does that by leading enterprise-wide initiatives aimed at creating a culture that passionately embraces modern engineering and organizational practices. These initiatives have resulted in the establishment of Verizon's Dojos and Verizon Tech Days (our internal DevOps Days); democratization of Verizon's technology decision process; creation of IT communities; initialization of an innersource program; and, adoption of Product Management practices and organization structure.


Previously, Jacki held roles as a business analyst and project manager at Accenture, AllianceBernstein & Goldman Sachs. Her passions include Product Management, Enabling Cultural Change at Scale, & Diversity and Inclusion. She earned her B.A. in Economics from Bucknell University.


Josh is a Senior Enterprise Architect/Evangelist at Verizon, enabling development teams to build the solutions that make Verizon the leading communications company in the U.S. He does that by being at the leading edge of technologies, adopting startup mentalities and solutions, and applying them at massive enterprise scale to transform the way Verizon does IT. He also pushes community building and culture change within Verizon as MC and content coordinator for “Verizon Tech Days”, internal conferences based on DevOpsDays.


Previously, Josh has been a software architect for multiple Verizon platforms used by thousands of developers including Cloud Foundry, Openstack, and Apigee. His passions are Enterprise Architecture, DevOps, & API First. He has a B.S in Computer Engineering from Ohio Northern University.

JD

Jaclyn Damiano

Senior Manager, Product & Program Ownership, Verizon

JS

Josh Stone

Senior Enterprise Architect/Evangelist, Verizon

Transcript

00:00:05

I'm Josh Stone. I'm a a it architect and evangelist within our cloud services and our DevOps platform engineering organization. And with me is Jacqueline Damiano. She's a people leader over our project managers and product owners in this space as well. Now, to get things started, before we go, I wanna say this transformation story that we're sharing with you here today, it's a very, we're transforming in a lot of different areas simultaneously. So we're really talking, um, at a broad level about everything that we're doing during this talk. So this is a perfect opportunity to come speak to us afterwards, up in the speaker's corner, where we'll be more than happy and enthusiastic to really go into the depth in each of the areas of transformation that we're gonna be talking about. So let's get, let's get started with a little bit of a video. Um, and this really kinda shows where Verizon's come from and where Verizon's going.

00:01:16

I was there in Chicago when Bob Barnett made the first commercial wireless phone call in 1983. Yeah,

00:01:21

This is Bob Barnett in Chicago.

00:01:23

We were both working on that first network that would eventually become Verizon's. Back then, the idea of a nationwide wireless network was completely unreasonable. But think about how important that first call was to our lives. It opened the door to the billions of mobile calls that we've all made in the last 34 years. Sometimes being first means being unreasonable. I'm proud that I was part of that first call, and I'm proud that I'm here now as we build America's first and only 5G ultra wide band network with unprecedented wireless capacity that will not only allow for phones to be connected, but almost everything transforming how we all live. Once again,

00:02:03

As you know, this call today is the first call that we've made on the cellular system.

00:02:13

All right. We, we sure have come a long way since that call in 1983. Um, just so from the audience here, who actually had one of those car based cellular phones? Show of hands? Anybody? I see a couple. All right. Who's got a smartphone today? All right. Absolutely. Who's got a Verizon smartphone? Yes. Excellent. That's what we like to see. Um, so, you know, Verizon's been a leader in this space for, for decades. Um, and one, we're really, really excited about what's going on with 5G. If we could cut back over on the screen to the, the presentation,

00:02:53

What we're doing, just a little bit about us. Um, we are a massive company, right? We, we have coverage over 98% of the us. Uh, we're a Fortune 20 company, over 126 billion in revenue, and over 150,000 employees are a part of our vte. Um, and like you just saw in the video, very excited about that launch of 5G happening just October 1st. We launched the world's first commercial 5G network offering. So a lot of things are happening at Verizon, but a lot of disruption as well. You've got a race to the market in 5G. You've got saturation in the, in the cell phone space. Everybody's already got a smartphone. The only way you can get new customers is by taking them from your competitors, or they're trying to take them from us. So it's a more competitive than it's ever been in that space before the entire home.

00:03:52

Entertainment, tv, broadband, internet, telephone, all of that is in a period of massive upheaval. And so for Verizon to maintain its leadership during this period of massive upheaval, Verizon's gotta be able to deliver value faster and more efficiently than it ever has before. And that's really at the core of what we're talking about here today, our digital transformation, our transformation to be able to deliver more value faster and more efficiently than we ever have before. Now, this is probably sounding pretty familiar to a lot of you today, um, as you've started to see all these different talks from other companies talking about different transformations and how to deliver that value more quickly, more efficiently. So what makes this talk different? What makes Verizon different in this space? Well, for starters, it's our scale, um, and not our scale in terms of the, the deploys of the commits.

00:04:48

Um, you'll see a lot of companies are doing that, um, a lot more than we are. But scale in terms of the number of applications and the number of different workloads that are out there running to support those applications. 2000 different IT systems and applications that it has ownership of. Those are managed by over 10,000 engineers and run on over 70,000 workloads. So as the picture kind of shows, Verizon is a pretty big ship to try and turn as part of a transformation, but scale's not our only challenge. Um, historically, Verizon has come about through the joining together mergers and acquisitions of a lot of different companies, doing a lot of different things in a lot of different ways. Um, so there's a lot of fragmentation and inconsistency in the application of tools, processes, and skill sets. So in a transformation, you've gotta treat all these different lines of business a little bit differently.

00:05:47

'cause they're all coming with different contexts, with different backgrounds. Um, and that leads to a wide variance in the developer skills. And we talk about those 2000 applications. Those are are not 2000 ideal candidate applications, right? We've got things that we've just built last week. We've got applications that we've been dragging along for the last 15 years. Um, so a wide range in terms of brown and Greenfield there for the applications. Add into that, we've got a very distributed engineering group. We have offices all over the us. A typical engineering team might have some of the team in New Jersey and Texas and Florida, and a good contingent in the team over in India or Peru as well. Add into that the regulatory compliance of being a telco. You've got a lot of challenges. And so to solve those challenges, that's what we're talking about here today. This is the approach Verizon's taking to get past those challenges and get us on the way to our transformation. And it's threefold, right? We adopt public cloud, we modernize our tech, and we transform our engineering culture.

00:06:54

That's the game plan. That's, that's, that's it. So adopting public cloud, what does that really mean for Verizon? For us, it means that we're moving the majority of our workloads up into the AWS and trying to move as many of them as possible. A large contingent of those by the end of next year, that's 29,000 workloads being moved up to the public cloud long term and targeting 20,000 by the end of next year. And while we're moving, we've gotta re-platform and make them cloud native and cloud friendly to do this. This is gonna be about 600 applications. When it's all said and done, get moved up there. Um, uh, when you say, all right, 600 isn't really half of 2000, you gotta look at, there's also a lot of rationalization that's happening on those 2000 as well, taking that 2000 and shrinking it. But when this is all said and done, when all the pieces have fallen, most of our workloads and mo and the vast majority of our directional workloads will all be up in the public cloud.

00:07:49

The public cloud is our future. Now, to move those 600 applications to the public cloud, that's difficult to do centrally. You have to train up each application team to now become experts in cloud. And one of the ways that we're assisting with that, one of the ways we're driving that is leveraging our dojos, right? Our dojos are immersive learning centers where team go teams go for six week engagements to learn. They bring their backlog with them and they learn from coaches, DevOps, agile, uh, product thinking. But now this year we've added an offering to it, cloud adoption, cloud migration. So we have coaches who are trained up in AWS, we're partnering with AWS itself to bring some professionals down to assist. And we're allocating 70% of our dojo's bandwidth on public cloud migration. So we're putting a heavy investment here in training up our, our engineers to be ready for the public cloud.

00:08:53

So that's the process. Once we get them up there, we still have more challenges. <laugh>. So 600 applications, you can't have 600 individual accounts, 600 individual VPCs with 600 private pipes back to your data center. It just doesn't scale like that. So we've gotta learn to share. And so we have shared accounts with multiple, many, many applications in them, but still trying to adhere to lease access principles. How do we get all of these applications to run side by side without stepping on each other, without accessing something they shouldn't be? And that's been a challenge. We've had to build a lot of tooling and processing above and around. So we replaced our console with our own ingress points, and we've also leveraged a whole army of serverless functions to go and hunt down noncompliant infrastructure and shut it down. And finally, the only way you get into production is through an automated pipeline using templates and images that have already been scanned and signed, right?

00:09:56

And that was the few minute nutshell summary of our entire public cloud strategy, <laugh>. But moving to public cloud is not enough in and of itself. We also need to modernize our technologies of the applications that we're moving up there and the applications that we're keeping in our data center. And when we talk about modernizing tech at Verizon, a lot of that comes to, um, rationalization and consolidation of duplicate and unneeded solutions. Um, like I said, 2000 applications coming from a lot of different business units. There's a lot of variance in the tools and approaches used there. Um, we, we use the, we use the analogy of we have a lot of snowflakes. No, no. Two applications are alike. And so the, the mantra of our modernization rationalization strategy is going from snowflakes to cookie cutter, right? And we've, we've tried this a couple different ways, and then we've had some successes and some failures.

00:10:51

I'll say. When we first started this, the first 30 applications that were coming into the cloud, we said, all right, you need to get to some common patterns. You need to get to some common templates. We don't wanna see 30 unique applications. What do you think happened? We ended up with 30 unique templates, <laugh>. So now we've built up additional, um, pipelines and additional tools to make it easiest for the developer to adhere to that standardization, right? What we're also doing is looking at these systems themselves. Can they be consolidated? Um, can we take the vast mesh of different systems that make up an IT organization's functions and break them into consolidated API based capabilities that literally lays the, the foundation for our API transformation that's happening as part of modernization. I'll say this is very early on in the process for us. We're just now at the point where we have the, the scalable platform in place and we have the, the processes and the tools there to now go out to our business partners and to our IT organizations and say, look, the tools are here.

00:12:04

We're ready for you to transform. 'cause transformation here is not an app by app activity. It's really going organization by organization and redrawing the interfaces and capabilities of each group looking towards the future, what's coming next in our modernization. We've had good experience in the past with PAs with it for our greenfield applications. Now what we're doing is taking container orchestration using industry best tooling, industry standard tooling, docker, Kubernetes, container orchestration, integrating it with all of our Verizon services, logging, monitoring, identity and so on. And then putting in front of it a full source to image pipeline that really takes, goes beyond just a, a container orchestration platform and into a full paths like experience, but built off of Docker and Kubernetes. Looking even further beyond that, can we get away from that infrastructure entirely? Can we go to serverless? And so we're ki in a, a mode of experimentation right now with serverless, where we have, I think one production serverless application.

00:13:12

And just recently we ran, um, hackathons with 70 teams to try and identify where are the pockets of value? What are we gonna go tackle next, um, in adopting serverless. Now, the last thing I wanna talk about in terms of our modernization strategy is how we're dealing with the platforms and tools that lie underneath the software that we're running. And we're really making an active shift to transition from being commercial cons, consumers of commercial software into contributors of open source. And when I say that it's one thing to take a corporate strategy to adopt open source, but at that point, if you adopt open source and then shell out a bunch of money on support contracts for that open source, you're not really taking the full full advantage you can of participating in that open source model. And so we, we've shifted our, our approach to leveraging open source where we can, but now we want to take the step beyond that, which is it's not enough to just consume.

00:14:13

We should also be contributing within it. As you've heard many times today, your people are your best resource. They're your most valuable resource. And so what this is here is an investment in our most valuable resource in training them up. We think the money's much better spent investing in internally on our engineers to become open source contributors than it is to spend on a support contract. And that's just starting to get into all of the things that we're doing around transforming the engineering culture and to take us the rest of the way on that transformation of our engineering culture. I'm gonna hand it over to Jackie.

00:14:50

Thanks Josh.

00:14:56

Okay, cool. So I'm super excited to be here today and talk to you about a cultural transformation that we have underway at Verizon. It's called Tech Mob. So very simply, tech Mob is about enabling and leveraging our most valuable asset assets, our technologists. In order to support this initiative, we've established three work streams, inner sourcing, evolutionary architecture, and chapters and guilds. Before we get into those specific work streams, I wanna talk to you a little bit about the foundations that exist for this program. So this program is rooted in open source principles. The first one, participation. Historically, Verizon has been a great example of Conway's Law. We speak along organization lines. We collaborate within our own VP organizations and within our own business units. Now we're asking people to collaborate to create better products. Rapid prototyping failure isn't something that I love talking about. I don't love to put it in presentations about how something went awry, and I really don't love it when my SVP calls me in to talk to me about my latest failure. But what we know at Verizon, or what we're beginning to understand is experimentation and fast failure is important to building great products for our customers. Meritocracy, historically at our company, people really optimized on delivering a project at a specific date because we said we were gonna do it.

00:16:40

Now what we're optimizing on is the value that we're delivering to the customer. We're making sure that the product that we're delivering is the right product at the right time. In a meritocracy, repos get contributions because people see value in that product. So the projects that have the most value get the most contributions. It's a true meritocracy community. I think all of us have those lone wolves and superheroes in their organizations who like to go off in their corner, design in awesome strategy and come out and try to save the day. But research shows us time and time again that no matter how good that superhero is, it's never gonna be better than getting a bunch of diverse people in a room to solve a problem and to create a product. So it's through community that we get really excellent products to our customers. Alright, so now I'm gonna take you through the work streams that I talked about earlier.

00:17:46

The first one is inner sourcing. So you may be wondering what inner sourcing is. Inner source is simply defined as open source within the confines of an organization. So at Verizon, what this means for us is opening up our repos, sharing with each other. Inner sourcing has a ton of benefits. The first is transparency. Instead of lamenting that we have a problem because of someone else's code repo, I now have ownership and the ability to go into that repo and make changes and to clear my own obstacles. It's so powerful. Reuse. How many of you have address validation code somewhere in your organization? Shipping, customer billing, et cetera. We have that at Verizon too. As you can imagine, we don't have one of these instances of address validation. We don't have 10. We have dozens in an industry that is experiencing extreme disruption in extreme competition.

00:18:54

We don't have time to be wasteful. Inner sourcing allows us to leverage things that have already been built and to not build them again. So reuse innovation, more people looking at code means that people are gonna have ideas that are different. So a great example of intersourcing and how it's been working at Verizon actually goes back to some of what Josh was talking about with our container strategy. So we don't have a formal organization that actually focuses on our container strategy. It's actually made up of 17 people who are super passionate about that technology and bringing it to Verizon. Those people have contributed over 60 pull requests that have been merged, and they're the reason why we have a great container strategy. Now, you might be sitting there thinking, and I know a lot of our lawyers have had these thoughts as well, gosh, it seems really radical. You know, you're blowing open the door on everybody's repos. This is kind of dangerous. And as a a code repo owner, you're saying to yourself, I really don't want those guys and women in my code. They're gonna mess it up. So I'm not suggesting that you do this without thinking about the parameters that you need to instantiate in order to make sure that you're protecting against that risk. At Verizon, we have inflight and at risk scans to make sure that our code repos retain their quality.

00:20:23

Okay, now we get to go into the second work stream. I was talking about earlier, evolutionary architecture. So at Verizon, in in in history, a lot of the decisions that were made about technology and the patterns and practices that we had were shrouded in mystery. Um, some people would even say that they had no idea where these decisions were made, and, and they lamented the fact that, that they didn't have the ability to make them themselves. Verizon has historically been a command and control organization. An engineer would ask their manager, who would ask a director, who would ask a vp, who would ask an SVP what tools they should use with evolutionary architecture turning that hierarchy on its head, we're asking individual engineers to stop lamenting that they don't have the power, and we're asking them to claim their voice. We're telling them that they're the people who are closest to the technology and they need to get in the driver's seat. As you can imagine, this is a pretty big cultural change for us.

00:21:25

So let me talk to you a little bit about how this process works. Firstly, we do define at a strategic level some of the big things that we wanna get done. So an example here is that we're a cloud first organization that is adopting AWS. Once we have that strategy set, our senior leaders are looking to our engineers to then figure out what comes next. So a good example is that someone would think that AWS Aurora should be adopted. We've spun up a GitLab repo that's accessible by 10,000 of our engineers. And you heard me right? We have 10,000 engineers, <laugh>. They all have licenses much to GitLab's Happiness <laugh>. Um, and they get in there and they, they uh, they talk about what tools they think should be adopted. It's within this form that they then debate with their colleagues. We have moderators who make sure that those debates are fact-based.

00:22:24

They make a decision after those decisions are make, are made. Say for example, we're gonna adopt AWS Aurora, we then bake them into enterprise enterprise tech stacks. These enterprise tech stacks are helpful, are helpful to our engineering groups to give them a model that they can then transform their organization into. Now at Verizon, as we've said, our scale is large. So sometimes our solutions aren't one size fit all and we recognize that. So once we've made a decision and we've created these tech stacks, we're then gonna go out to our portfolios to see if they fit. If they don't fit, we're gonna figure out why. Is it just an idiosyncrasy within that portfolio or is it something that should be applied more holistically? If it is, we're gonna learn and we're gonna go back and modify those tech stacks. So that gives you a little bit of an overview into our process. Alright? You can imagine that inner sourcing and evolutionary architecture would be difficult to do in a command and control, very hierarchical organization. So what we did is we went out and did a ton of research to figure out how we were gonna organi organize ourselves in a way that allowed us to be more nimble and allowed us to collaborate across business units and VP lines.

00:23:47

As with a lot of research, we found a lot of things <laugh>. We found communities of practice, communities of interest, centers of excellence. None of these things felt like the perfect fit for Verizon. So what we did is we took an amalgamation of the attributes of those things and we came up with chapters and guilds. We actually borrowed chapters and guilds the, the terms from Spotify, but they mean something very different to us at Verizon. Chapters are formal constructs. These organizations are made up of people who are experts and practitioners. They advise throughout the evolutionary architecture process and they also serve as evangelists for the tech that we decide we're going to adopt. Guilds are informal, they're organic. You need not be an expert or even a practitioner. All you need to do is have an interest in that particular area. These folks will get together, they'll come up with code snippets, they'll share them, they'll come up with best practices, make sure that you know, folks have access to them.

00:24:53

So you may be sitting there wondering how we decided which chapters to instantiate. A little while ago we had our architects get together and map out our technology and practices at Verizon. We then went through an ization process and we identified 12 communities that we wanted to establish. We then went out to our leaders and said, we need experts and practitioners for these particular chapters to staff them. So they came back and we have about 250 chapter members right now. So we've been able to mobilize these chapters. Some examples by the way, include infrastructure, middleware, emerging tech, et cetera. We've been able to mobilize them, they're active, but in, in, in all honesty, we're struggling at this time in terms of how they engage and having them show up. And we're struggling for a lot of the same reasons that I bet you all, um, might experience as well.

00:25:49

We heard Scott Pru this morning talk about the, uh, the Brents in his organization. We have a lot of Brents <laugh>. So this is something where we're having to make time for our employees to be able to engage in this type of process. We're having to help them understand why this is a priority. Essentially, if they're not engaged and involved, it's like not voting for in in government terms. You're not at the table, you don't get a voice. So the decision is being made for you. So what we've done is we've worked with leadership to carve out five to 10% of a person's time to serve on that chapter, which has been helpful. So we're working on it outside of all of these things. We're also doing a lot to invest in our engineers. So we have two certification programs right now. I'm proud to say that over 1500 of our engineers have been certified on the AWS platform. We also just recently spun up a container program and that's generated a lot of interest.

00:26:56

So we're not without our challenges. I've been with Verizon for about seven years and before that I was with a lot of different companies in a few different industries. And what I've seen at Verizon in the past two years under Ross Clan's leadership is that we've really been able to move the needle. We had a lot of this technology in place before, but what we were missing was the people component. And we are missing them adopting these things in a passionate way. And you can't fake that. You can put as many tools as you want into the environment, but unless your people are passionate about them, it's not gonna work. So the scale of effort, getting people to do something at the scale that we need them to do it is really hard. Having 10,000 people do anything is a big challenge for us. So much so that we actually hired a marketing and communication and communications team within it to help us describe to people what's in it for them.

00:27:55

Um, and making these changes. That's driven a lot of engagement and motivation. We talked earlier about the Brent Syndrome. A lot of people with a lot of things to do, but we're trying to articulate to them why it's important and for not only Verizon, but for them to upskill and stay relevant. There's lots of change on any given day. An engineer at Verizon can receive communications about cloud migration, serverless, containerization, and it's overwhelming. So we're trying to work to figure out how to feed this to our engineers in a way that isn't stimulus overload. As I mentioned before, Verizon, we get it done no matter what, but sometimes that becomes a fault. We're trying to encourage people to slow down to speed up. Sometimes you need to take a second to learn new tools in order to get the longer term goal. Alright, and this is the infamous what we need from the community slide. So we have two asks. We're new at intersourcing, we're new at these communities. If you all have any experience spinning them up in large scale enterprises, we love to hear it. We'd also love to hear it. If you're not large scale, we think that we can learn from everybody. So as Josh mentioned, we'll be around this afternoon. We'd love to hear from you guys. Thanks so much for your attention.