00:00:02

Next speakers. I had the pleasure of meeting Jonathan Moore, chief software architect at Comcast earlier this year. And it was amazing to hear his nearly decade long journey helping transform Comcast, which currently generates about $100 billion of revenue annually for a very long time. Their primary business was running coaxial cables to people's homes, but over 10 years, Johnson has been part of an ambitious initiative to create a world-class software engineering organization. Now employing over 8,000 software engineers. I asked him if he'd be willing to share that Comcast story here at DevOps enterprise asking if he could cope, present from someone within a major business unit that could describe why John's contributions were so important for reasons that Ranga the VP of entertainment technologies that Comcast will describe. John is unable to join us today, but in his place, Jessica Sant VP of discovery engineering will be presenting in this place.

00:00:55

By the way, on a personal note, um, a couple years ago I got a Comcast gigabit service and it was one of the most amazing customer experiences in my life. I love the service so much. I actually asked them, can I talk to someone who matters to gush about how amazing this experience was, and ended up talking to his manager who told me about the, all the work that Comcast is doing to transform the customer experience. And I can personally attest, uh, that it seems to be working, bring it, uh, please welcome the team from Comcast.

00:01:35

Well, thank you very much for having us, uh, gin, what a warm welcome. And I think everyone all's gin, a great round of applause for organizing this great conference.

00:01:50

We're equally excited to be joining you here. Not just because we think our story is more interesting than anyone else's. It is just as interesting as everyone else's. We're excited to be part of the learning community that is giving as much back to us, as we hope we can give to you, uh, for our journey and for our story, we're going to break our story into three pieces. We're going to tell the story of how a 50 year old company incubate an idea brings it in-house and then use it to scale up as it confronts new challenges and imagines itself as a platform company. For those of you that do not live within our footprint or serving area, Comcast is now a global provider of broadband services, as well as entertainment services. And those entertainment services are video and theme parks, not only here in north America, through our footprint as Comcast, uh, or through syndication partners, such as Cox, but also in Canada where service like Rogers, Shaw, and Videotron leverage our technology to entertain their own customers as well. And then since 2017 with the acquisition of sky are now becoming more of a global company. I am excited to be joining our presentation here with our chief software officer,

00:03:23

John Jonathan Moore.

00:03:26

So, um, as you heard, Jean mentioned, John is unable to join us. Um, John is the coach for his son's high school ultimate Frisbee team. And in John's words, a 40 year old man should not quote, show him how it's done. Uh, so John had surgery on Monday to repair his very shredded Achilles tendon. Um, so he is at home watching us on the live stream. So everyone wish John, uh, very healthy, good vibes, and please don't wish us to break a leg. So my name is Jessica Sant. I'm the VP of discovery engineering at Comcast, um, and get to step in for John here. Um, luckily I was part of this journey from the days I'm so excited to be here

00:04:10

Enough about us. Now, this is what we looked at. Like when we started, as you can tell, my hair was actually black, it's gray, even with dye has, has gone purple. Um, and as we relate a story, maybe, you know, that transition would make sense. Um, so 2016 around the time that I joined and just joined you joined shortly after we 2006, I'm just trying not to age myself. Um, 2006, we had one video product, right? So across our footprint in north America, with one video product, we did not build that video product, right? That video product was delivered to us by set top box manufacturers and to minimize risk would have multiple set-up box manufacturers, provide equipment to us, that'll provide into your homes. And each of those would build their own version of the guide. We were integrators and we're an operations firm. We were not a product development for that leverage software dev ops and agile.

00:05:20

Yeah. So this was a really big waterfall process, right? As, as Ranga was saying, we were an integrator and an operator. Um, and, and as with any large company with big waterfall and standardized processes, um, they had names and they of course had acronyms. So we followed something that was a said process. Um, and this is basically the idea where you had to go and present your idea of what it was that you wanted to build to all of the people in the big purple robes, who would then bless your idea and say that it was something worthy of being created. Um, and another part of the said process was this thing called a Jad session, a joint application design. So this is where you would talk through all the interfaces that needed to get built and use cases, not user stories, but use cases full of UML diagrams, defining, you know, what part was going to talk to what other part. And so, you know, we had sad sessions, said sessions and Jad and Jad sessions. And of course, one day while John was in one of these meetings, he accidentally referred to it as a sad meeting, um, which got a couple of cross-eyed looks and he quickly corrected himself, but it was kind of, um, a very, uh, key Freudian slip of just kind of how all of this process made him feel.

00:06:35

So, so let's reset. Right. Um, if you've been following the general sessions, you've listened to great presentations by John Deere. You've listened to just now BMW and, um, comparable, comparable compute where yesterday and the key theme seems to be recurring that we are touching to that you have this well-established long lived company that in the past 5, 10, 15 years discovers that it's under strategic challenge and that a way to transform it's going to require an adoption to allow it, of technologies and processes and culture that allow it to be more agile, but just resetting 2006, it took us a whole year to release a new feature, right. Just, we were releasing at a sort of cadence of about a release a year and from start to finish from ideation to launch was three years.

00:07:32

Yeah. And, and in fact, right, like the way that you progressed your career and the way that you showed value is you were really good vendor manager.

00:07:41

Yeah. And I think, uh, I think it was, it was, I think in early then it was, you know, the title of chief process officer was actually the much more important than chief product officer because we did not have a product. Right.

00:07:55

Yeah. And so we were, you know, we were really that integrator and operator, but we realized we wanted to kind of make ourselves into a technology company because that's how we needed to, well, that's what we needed to do to compete. And so this story that we're going to tell you is really about kind of three different groups of people and, and that you will probably hopefully recognize these three groups of people in your organization as well. Um, it's about the practitioners. Practitioners are the folks who are, are building the code and writing the software and, and, and doing the work there's champions. Maybe people, managers, maybe tech leads, things like that, but they are the folks who are providing cover, fire, their help giving direction. Um, and then there's the investors. They're the folks who are giving you, you know, who are, who are, have some of the visions and they've got the business cases and they're, they're the ones giving us the money.

00:08:37

So really those practitioners are helping to define these best practices and, and writing the code and giving that over to the champions. The champions are kind of on the hook to deliver results to the investors. And the investors are on the hook to give time and money back to those practitioners so that they can, they can do the work, but really, you know, it's a really virtuous circle and you need all three of those parts working together to be able to make any kind of transformation. And I would argue that there's a whole lot of trust. That must be, must be built up between those three groups, three groups, but also you have to earn that trust and kind of, and show each other how you're going to work to be able to earn and maintain that trust. But culture is a really strong thing that backs, um, that backs this whole thing, this whole process.

00:09:24

And I think the culture that evolves to allow transformation to happen, um, is really important, but you also need to have a base culture to even make that possible to begin with. And I think we were very lucky in that the culture that we had with M Comcast really embraced the idea of change. Comcast is a 50 year old startup. We started off in Tupelo, Mississippi, where Ralph Roberts bought a, you know, bought a pole, right. Um, and then we've grown and grown through acquisition and through, um, building some really great technologies, but having that fertile soil was really important

00:09:58

As you think about what we've done. Uh, and I think, you know, when, when you think Comcast, you think hierarchical well-established with wall street recognized firm. So the first thing that comes to mind is maybe this was a change management initiative. No, it wasn't. It was a succession of successes, right? We initiated, uh, an incubated, a company called Comcast interactive media, which became the proving ground, uh, in 2006, for our ability to build a software based product, Comcast interactive media or SIM, as we call it, refer to the internally was actually initially outside the corporate headquarters, separate space, separate facilities, independent culture, and that culture was allowed to grow until it was ready to bring in. And I'll share one story just within the spectrum of culture and trust, which was when we needed to sort of expand that office instead of doing what a hierarchical company would do to sort of find the highest, I don't know, uh, facilities, uh, person in, in, in the organization, and then trust them with the role to do that. It was principal engineers that lived in the target office locations that did the office search. And this was yet another example and we share more vignettes along this thesis of we, all of us assembled here have access to all the processes, all the documentation, all the technologies associated with agile dev ops transformation. But we believe that what we can share our, the internal sort of conversations, the demonstrations of trust in an authentic and sustained way that ultimately has sort of carried the story for us.

00:11:51

So we decided that we to change the way that we worked go from this year long waterfall process and being this integrator and operator to building software ourselves. And this was, this was 20, you know, 2006, 2008, 2010. And so we bought the bulk on scrum and we read it, cover to cover and figured out like, Hey, this is the process that we're going to, we're going to drive. John became our first scrum master and he taught our colleague Matz Lesko also how to be a scrum master. Um, this is actually a photo from back in the day. Um, you can see John and Ranga, they're looking a little bit younger.

00:12:24

Yeah. I think my daughters said that I used to dress like a retired detective and

00:12:32

I

00:12:34

Took care of that. So I think we're good.

00:12:37

So, I mean, you can kind of see from this photo and if any of you were doing scrum back then as well, right. We've got an Excel spreadsheet up in the corner. That's tracking all of our user stories. We've got sticky notes, we've got post-its and all of the hallmarks of a good scrum team. But the other thing is that's important in this photo is that it was a very cross-functional team. We've got Ahmet from QA, we've got wronger from engineering, we've got Moe from engineering. We've got Jason who's the back of the head, there is our product manager and ops is back in there as well. And having everybody involved in part of that process so that everyone could feel a part of it and they could learn to trust the process. And what was going on was really important part of this journey.

00:13:17

I think, as you look at that picture, this is about 2007, two thirds of the people in that picture. And this is generally true of people that started in this journey are still with the company. So as you think about an engagement measure, uh, you know, we were talking yesterday in the people at dev, uh, um, uh, forum about EMPS, about, you know, just having these well measured sort of conversations. Another engagement figure is, do you keep your people, are they part of that journey of a period of time? And we're really proud to be able to say that, you know, we we've sort of been as successful as we can be.

00:13:50

And we've had a lot of boomerangs actually Ranga was a boomerang. He actually left us for a while and came back. And I think, you know, that is also a Testament to the culture and everything that we've built. So even though we were working in this very cross-functional team environment, we still had the Neapolitan ice cream of organizations, and that we had a dev group that reported up to a dev manager and QA group that reported to a director of QA and an ops group that reported up to a director of ops. And so even though we still had these handoffs between these groups and some of them were, you know, throwing things over the wall a little bit, we were working closer and closer together. But so we, even though we lived in this structure, we were still moving and keeping everybody involved, but living in, you know, in a very,

00:14:29

Yeah, I'll tell a story of just back in 2006, the way to think about how we were all guys before we were even a software based product development firm is every button on your remote was essentially a VP associated with that. Right?

00:14:48

So a couple more photos from back in time. So this is one of our grooming sessions and our estimation sessions, right? We were doing planning poker. And so you can even see John there with his poker cards, that he was showing everybody how to do an estimate. But the really interesting thing about this photo is that over here in the lower right, is Amy bans. And next to her is Sam Schwartz. So Amy was the head of Comcast interactive media. Um, and Sam was her right hand. Amy has actually gone on she's now the head of Comcast ventures and Sam is our chief business development officer. But in this meeting, right, we were involving, you know, the practitioners, the champions and the investors in this process. And so at the same, at the time that the engineers were throwing out an estimate of like, Hey, this is a five story point of effort, Amy and Sam were throwing out a business value estimate as to how important they felt this feature was.

00:15:38

And so between those effort estimates and the business value estimates, it helped us to prioritize the work. And we don't, we aren't not doing that process anymore, but it was really important for the time to involve our investors in the process. So they could see what we were doing and why, and it wasn't that the engineers were going off and doing this whole process. And that was crazy folks doing whatever, but they were part of it and they could see the value in it and they felt involved. And so then a month later when we had our first sprint review and we were delivering new functionality and new code, they could see the process working and they, and because it was very transparent and we were in, we tried to keep everybody very closely involved. Yeah.

00:16:15

And two things to note in that picture, I think the first one is somewhat tongue in cheek. One way to get key executives in a developer meeting is to give them a lot of candy and this is going to see it between Sam and, uh, and, uh, and Amy, there's a bag of candy. So that's how we pull that off. But, but the second thing, which is just as important is that at, even at that point in time, right, Sam and Amy led Comcast interactive media and Comcast interactive capital, right? These were outward focused investment arms. They still exist today. They do a large book of business, right. But think about a company that is leaning into how technology based products and innovation are done and that doing that at the level of scrum of sprint review participation, right? So if you can get that level of engagement and the culture is really reinforcing the support at that level, that allows you to attract incredible teams of people.

00:17:20

And just thinking about trust a little bit. When we started, we had a product that SIM was building called cast. Um, and, uh, we had a decision to make, right. The organization, Comcast had an incumbent search provider. It was a commercial of the self, uh, platform, which was functionally better, but objectively, uh, sub performance relative to opensource. And we challenged as SIM the proposition that we should use the already licensed version and rather invest in the open source, increase its functionality so that it was a feature parody, knowing that the payoffs would be worthwhile. And so this is effectively our first open source contribution, uh, you know, that, that we funded. And as you fast forward into 2019, we now have an open source program office that is actively and permanently resourced to take innovation from our team and bring it back into the community.

00:18:24

So this was our fan cast website. So fan cast was an entertainment website meant for a national audience. I think we got this from the way back machine and, and one. So it was basically a place where you could go to find out what's on TV. And what would you like to watch and interesting factoids and things like that. One of the pivots though that we made is that we realized that we could super serve our Comcast customers by giving them an even deeper experience into entertainment. And we had this crazy idea and I'm sure we weren't the first, but I think we were the first cable provider to do it, of allowing customers to stream TV shows so that they could watch their movies and TV, you know, on their TV, but also on their, on the web. Um, and so it was a project called ODL on demand online.

00:19:08

And so we kind of realized that this was going to be like our next big thing that we needed, that we all needed to focus and work on. But because we are part of a large company that has a lot of processes, we were still being asked to go through the said process. And the first part of that was an order of magnitude, which is basically defining your level level of S a level of effort, right? Two person months, five person months, whatever it was. But we all knew internally that this was the highest priority that we needed to work on. And so we kind of felt silly that they were still making us go through this, this order of magnitude process, but so be it. So, but as again, as any kind of good process has, there is an acronym and this one was called oomph.

00:19:55

And so John was on this zoom call and there's like 50 people on the phone and they're going around the horn asking everybody for their estimates and Hey, this is five person days or 10 person days. And they said, SIM, what's your estimate. And John just kind of side and said, we'll get it done. And the person on the other side of the phone just kind of took a, took a, took a beat, and then said, how about I write down that you're committed. Right. And I think it was a big, you know, I think sometimes transformation starts with a small act of rebellion, but also it was a big thing for our, um, our program management office to kind of be okay with that and kind of acknowledge that we were going to be doing things a little bit differently and you know what, that's fine. I'm not going to force you to give us an estimate for our process. We'll put you down that you're committed, you guys do your thing. And so there's a lot of that trust. And a lot of that give and take that think we all need to kind of figure out how to work in

00:20:49

Was built. I think in a lot of those participate in meetings with senior leaderships, where everyone was on a first name basis and, you know, words that were not otherwise sort of prescribed business speak were being used, but in a safe environment that sort of just really reinforced that you could, uh, in right conscious make decisions for the business that you know, were objectively the right decisions. Definitely.

00:21:14

So, um, around 2010, his SIM was actually, uh, required by Comcast. And we became a part of part of the rest of the company. And instead of them keeping us off into this own little, like a little incubator spot, like they had, what they did is they kind of like mixed us all into, into the company while still maintaining our culture and our values. And so they took some of the SIM executives and moved them over to some of the other groups. They took some of the Comcast cable executives and moved them over into SIM. And it was a really great way that we could take the lessons that we learned and the culture that we created and help infuse that into the rest of the company. And I think that has been a big part of, of where we are today. What's one of the things that's really neat to look back on my nine years that I've been there. And again, if you told me nine years ago that I was syllabi at Comcast, I would think that you were insane, but I truly, truly love it. Um, it's really neat just to see how far we've all spread and, and how many different parts of Comcast that we're in and still taking all the lessons that we learned here and bring them forward in the company.

00:22:11

And I think that period bookends a moment of transition from 2006 to 2010, when we recognize that we could really internalize and operate under a sort of, you know, software base, uh, paradigm, uh, and that, you know, we could actually now make successive investments, uh, aimed at owning more of our, uh, on getting more control of our destiny. Uh, and the first thing that we decided to do was to become a full stack development company, right? Instead of, even up till that time, as much as we're building consumer facing applications, we're doing those on other people's platforms. And with RDK reference design kit, which is a consortium that runs in an open style, opposite style format, and there's about 400 licenses. We decided to own our destiny as close to the metal as possible. So no longer were we buying up boxes with proprietary operating systems, RADA we initiated. And the principal called me to us to an MSR project that has this, uh, real-time Linux distribution. That 400 licenses are part off, and now we can control innovation throughout the entire stack. And the story here is not so much a brag about owning the stack, but really understanding that to truly innovate. And as competition becomes more intense in this space, we need to sort of be agile across the stack that, you know, is a dependency on the work that we do. Yeah.

00:23:46

So then it was the Dawn of test-driven development, right? And, and we continued to involve to evolve our processes. And so instead of having that Neapolitan ice cream of dev QA and ops is we started to merge together dev and QA, and actually one of my colleagues, Tom Polacco and I, I was the director of engineering for Sam. Tom was the director of QA. We actually put, put together a proposal that Tom should no longer have a job. Um, I mean, he should still have a job, just not the job he had. And so we actually proposed and got it, got it through that. We shouldn't have a dev Oregon, a QA org, but we should have engineering teams that report up to the same engineering manager. And it was a really great process to be a part of along this time is when con continuous delivery became a really interesting thing.

00:24:32

And so one of our groups, the platform API team decided that this was a big goal that they wanted to hit, but they still had huge delivery targets that they needed to meet of new features that they needed to get out. And so one of the things that they were very conscious about doing was saving a little bit of time, every, every, every sprint to do work on their CICB pipeline. One of the things though that John had to do as a champion was that we also had a change control board, a CCB process. And so, although we've made the company comfortable with going from a one-year release cycle down to a two to three week release cycle, they were not comfortable with the idea that we were doing. We could do daily releases, or even our early releases, the idea that we could actually still maintain quality while doing things at that cadence scared a lot of people. And so what John did was basically to find out what are the things, what are the artifacts that they needed to be able to, for the CCB to be comfortable. And then we built those into our process as well.

00:25:27

And you know, now we're in the period 2010, 2014, a lot of teams have adopted agile. We've got a lot of products out in the market and every team has its own agile process, right? And we started observing a sort of set of scaling limits, right? Each team was adopting practices, some of which could benefit teams more broadly. Uh, so we decided that the elements of the way that we developed and released product that perhaps we could consolidate. And first amongst them was our CIC D pipeline. And by the way, this was not so much an imposition as it was, you know, a developer led initiative to really find the most sort of developer experience enhancing choice, uh, in this case. Uh, but the key problem we're trying to solve was as follows if we are to adopt, which we have been doing a sort of, um, similar to the Spotify organizational model, where you have cross-functional teams that can independently deploy one of the trade-offs you get with all that innovation is maybe you get, um, uh, you know, lack of compliance to things like security, or you don't have uniform test coverage.

00:26:41

So consolidating around a common CICB platform in this case using Comcost allowed us then to layer on all those, uh, as part of that standard deployment, including things like AB testing, of course, and more recently, we just launched a product called flax. So if you're a Comcast subscriber, you're familiar with X one and stream flex is really positioned, um, as a participant in this competitive space where cord nevers and cord cutters, uh, looking to alternative ways to sort of aggregate over the top content. The reason we bring that up now is that flex embodies in the way that the project is organized, the teams work to absorb and inherit all of the lessons that we have learned in the process of our agile adoption and our dev ops construction as an org. Yeah.

00:27:35

So a couple things to talk about our culture and kind of what made it possible. So I think collaboration was a really, really key point that made, made this possible. Um, here again is another shot of back in the day. This is actually from one of our war rooms. Um, I am happy to say the company is not quite as white and male as this photo seems, seems to, to make it seem. Um, but, but the point here is actually that there are folks from every different group here in the room, just trying to get, to get work together, to get it, to get it done and fix the problem, whatever was going on wrong in production that day with the two folks at the laptop, I'm guessing this had something to do with the community chief he'd go, um, and, and some of our databases at the time.

00:28:16

But, um, but yeah, so you really being the key point being collaboration is a really big key. Another, another important part is sharing credit. So, you know, we showed you this picture before with John and Ranga, well, this gentleman in the back, his name is Matt is Lesco. Um, and one of the things, so Matt was our VP of engineering. Um, and one of the things that he always made sure of is that during a sprint review in front of all the executives, that the team was, you know, the team was giving his, he always made sure the engineers were the ones that got to do the demos and to get to talk about the product and do those presentations so that they could get that experience, but also get the credit for the work that they did, but it worked out just fine for Matt to Matt is now our chief technology officer. Um, he, you know, part of that culture of sharing credit, um, is something that, um, that I think that we, we try to do a lot of today as well.

00:29:09

And executive buy-in is also another big, big part of this. And so this is a story that John told me where, so one day they were, there was a big all hands meeting and John and his team were of course running late. So imagine this big room kind of shaped like this one is, um, where there were screens awkwardly on either end of it. Um, and tables in the middle. There were all facing the one screen. Um, and there's, you know, John gets the meeting late enough that like all the tables are filled. There's people standing around the sides and, but there was this giant open space in the middle. And so John just decided to like take a knee and sat in the middle with his engineers and, and, and, you know, watching the, watching the, the all hands presentation and, and a gentleman, very nicely dressed in a suit similar to my dear friend Ranga comes and sits down with him on the, on the ground.

00:29:58

And, and John's Washington 3g station. He kind of like nods at the guy next to him. And, and, and, and John's wife also named Jessica happened to be standing on the side and she kind of leaned over to the person next to him and said, I don't think John has any idea who that is. Right. So the, the person up front is speaking and talking and said, yes. And I'd really just like to thank and introduce our CEO, Brian Roberts, welcome to the stage. Um, and Brian and Brian, of course, then Stan stands up from the ground sitting next to John. And so lesson number one is always know what your CEO looks like. Um, it's much easier to do in the time of LinkedIn and all of those things, but that is lesson number one, lesson number two though, is I think really a lot about, so John got, sorry, Brian then got up onto the stage and he spoke about how important he felt.

00:30:43

Sam was in the investment that they were making to help this, this team transformed the company. And one of the things I will say is having that support from Brian all the way, you know, all the way down made this transformation possible. And in fact, if any of you come to Philadelphia anytime soon, um, please please come over to our new, new building. So Comcast center was the tallest building in Philadelphia, but we outdid ourselves and built the Comcast technology center next door. Um, it is, it is a hundred percent engineering in the building and it is an absolutely stunning, stunning building. And it was, and Brian was heavily involved in the building of the creation of the building. Um, and it was basically him showing very much that we are a technology company now and making this huge investment in the engineers, not in the executives who stayed back in the Comcast center. Um, and I think that was, you know, having that buy-in from the top is huge.

00:31:32

Yeah. A big contrast from, you know, 2006, where we were in some random office space to now, there's like a Pantheon for developers, you know, 8,000 plus strong and growing. We are hiring and, you know, 30% we mentioned the original, uh, pay-TV application, interesting lesson from product here, which is as much as we have wonderful, you know, great world-class video playback experiences, 30% of our customers still enjoy that original guide, right? And this is through no fault of our own, trying to replace it with X one. It's because your customers will occupy a distribution. It is worth honoring that agile and dev ops is a really key strategic piece where you think about product market fit, really having those conversations with the customers and figuring out where those resources should be set aside. And with that, thank you for sharing with us, the story of the role of authenticity and trust in your agile and DevOps transformation. So happy to be here, looking forward to learning. Well, thanks. Thank you.