Connecting Comcast and SKY: Using the Dojo Format with International Teams

When Comcast acquired SKY Communications in 2018, there was sure to be an interest in teams working closely together on both sides in order to explore which products and services would work nicely together.


These explorations were driven largely by senior leadership and focused on big ticket, highly visible products and projects.


But when the DNS teams from the two companies came together last year, it was not because a senior leader asked them to connect. Instead, it was a love for DevOps! Two members of the SKY DNS team were invited to speak at Comcast's internal DevOps conference last year, and they were greeted by our Comcast DNS teams who were eager to work with them on a shared DNS Management System.


While the motivation to work together was evident, the distance and time difference between London and Philadelphia was not an easy obstacle to navigate. We decided to try using techniques from the "The Dojo Handbook" and the results were fantastic! We were amazed how, even as remote teams, we were able to benefit from immersion, micro-sprints, a working charter and constant feedback loops.


We are going to share our story on how these teams decided not to wait to be tapped by Senior Leadership in order to work together, the issues we overcame working internationally, and the success we achieved using the Dojo approach to coordinate.

NP

Nisha Parkash

Online Compliance and Brand Protection Manager, Sky

MW

Michael Winslow

Director of Engineering, Comcast

Transcript

00:00:08

Good afternoon, everyone. I'm Michael Winslow from Comcast. And today we're going to talk to you about how Comcast and sky were able to use the dojo format to work together. Even across the seas. I'm joined here with Nisha Parkash Nisha, do you want to say hi?

00:00:22

Hi everyone. Um, as Mike said, I'm Nisha podcast from the sky on the online compliance and brand protection manager there.

00:00:30

Great. So why don't we jump into our presentation? So you did give a brief introduction. So why don't you just tell people a little bit more about yourself?

00:00:46

Sure. So I'm from London, born and raised. Um, obviously loved the city. Um, I grew up in south. I now live in west London. Um, I I'm a big fan of tea. I think most British people are. Um, I think every task from a British person starts with a good cup of tea. So, um, I had to get that right. I went and took a proper British brew tea making class, uh, to get that right and just showing a little tip. The, uh, the idea on a good cup of tea is put your tea bag in the hot water. Don't touch the hot water, don't touch the tea bag for two and a half minutes and then put your milk in and you'll get a good cup of tea. Um, and another fun fact about me completely obsessed with kitchen gadgets. Um, I try and find anything available to make, uh, food look fun. And I think the best thing that I've found so far is turning a mango into spaghetti.

00:01:43

That's cool. That's cool. All right. A little bit about me. So, uh, I'm from New Jersey, a little town called Mount Holly, for those of you across the ocean in London. Uh, I know a lot of you are familiar with the Newark airport. I'd say probably about an hour away from there. Um, my other, uh, fun fact is I spent a semester working in Amsterdam and there, uh, you know, I want it to be able to order my favorite burger king meal. And so I learned how to do it in Dutch. And so really quickly, that's a, Zander Zander, Mio with, uh, met Coca lights, eh, jumbo fruits. So it's a little thing that I picked up along the way. And then, uh, on this day, uh, I would say that it's very important that to say that I support black lives matter. I hope you do as well. Uh, it's a very important movement and, uh, I'm very, it's a very happy time, uh, right now that so many allies are stepping forward and I really appreciate all of you.

00:02:45

Okay. So our story starts almost exactly a year ago today. Uh, I traveled over to London to visit the sky UK offices with Nythia rough, uh, for a tech woman talk. And, you know, I did not know that I was going to be able to meet the DNS brand protection managers over there, which happens to be, you know, very close to what my team does with DNS. So it was great to insert serendipitous to actually meet that team. And so here you can see basically what it was like over there, amazing offices, the Simpsons, quickie marts, a bus stop for the shuttle bus. And then, uh, we took this picture a year ago and it's so appropriate today, which is, you know, our differences are only skin deep, but our sales go down to the bone. Okay. A couple months later, uh, we were actually able to invite initia and, and Dawn Shackleton out to Comcast where we were having our, uh, Comcast DevOps event, our fifth year event.

00:03:44

And it was so great to have them out there and be able to get on stage and talk to Comcast. There's, it's such a huge part of dev ops to share, and they were able to come and share it to our entire community, uh, in Philadelphia. Um, another great thing about that, uh, October event, gene Kim was the keynote speaker. It was so great to have him talking about the Five-O ideals. It was just before the unicorn project came out. So it was a real exciting time and we were glad we were able to get them over there, very personable. And he talked to everybody and participated the whole day. It was a great, great event. Okay. So the conference was nice, you know, we we're, we, we are doers. I can tell you that much about Nisha and I, right. So while we had a great time at that conference, the media question we had afterwards is like, how do we work together?

00:04:28

And so we were like, how about we take the application that, uh, my team is in charge of and see if we can get it working at sky. So can we get vinyl, DNS working at sky? So let me just quickly tell you what vinyl DNS is. Since we mentioned it, a couple of times, vinyl, DNS is a zone is a DNS management system. And it's very important within the walls of Comcast because we have the second largest DNS footprint in the world. So without giving exact numbers, you know, we have over 200 and thousand zones that we manage and we have over 40 million DNS records within those zones. So very large footprint, um, with something like that, it's great that our application is able to enforce standards, uh, for all the DNS entries. Uh, we're able to give fine-grained access control to individual groups, which is very, very important, um, in this industry to not have it all flow through a single entity.

00:05:24

We also provide a robust API for integrations, so that people who do not want to use our front end portal, there's a headless option to be able to put automation in place, which is huge in DevOps. So we're thinking why wouldn't sky want to take a look at this? Because when you think about an application that needs to be able to support, um, a DNS footprint of this size, there's only a few number of groups that can actually take advantage of it. But I would say that, uh, sky was definitely one of those groups, right? Nisha,

00:05:53

Definitely.

00:05:56

So there we go had what we wanted to do. We wanted to basically get vinyl, DNS working at sky. So that's great. So let's just do it. Let's just give, uh, access to Nisha and she can start working on it. Well, we found out that we had a little bit of a problem there because while we had out all of our applications and our communication abilities, uh, for Comcast in Philadelphia, you know, slack channels get hub, uh, our application, our internal cloud, we found out that sky was just didn't have the ability to connect with us in those ways. So even something as simple as being able to come together in a slack channel, we weren't able to do so that made it very difficult. So we had to figure out how are we going to actually make this happen? So I guess what we thought was after thinking about it for awhile vinyl, DNS, we realized was open source.

00:06:47

We actually took the step of making sure that this application was something that we gave back to the community. And in this case, it helped us so much because now we realized the firewalls mean nothing at this point, because it's out in the public internet and public GitHub. We can just treat it as if we're two different citizens that wanted to use vinyl, DNS, and then just work with the team to do it. And so, while we couldn't get to our own private cloud internally, we found out that sky Nisha and the team over there were users of Azure. So we thought this was an exciting challenge because we actually never put our application out on Azure before. So, uh, we were really ready to accept this challenge. We were a go, okay. And so for anybody who wants to actually visit and, and take a look at vinyl DNS, you can go to vinyl, dns.io.

00:07:37

And from there, you can link to our GitHub account and see all of our documentation. Okay. So now we're talking about the dojo. That's, that's the way that we want to come together. So what is a dojo for those who don't know already? Uh, the Comcast dojo is an upskilling program and it's designed to be an immersive learning experience where full-stack teams come together and learn modern engineering product and agile practices. Um, so a lot of people in the dev ops community are familiar with dojo's already thanks to the pioneers before us, the Ross Clintons of the world, uh, the Jackie . Um, so let's put it this way. What we want to make sure that we come across is when you step into a school, right? You kind of know that your reason for being there is to learn. And when you step into an office building, you pretty much know that your reason to be there is to work.

00:08:29

What's unique about the dojo is when you step in, you should know that you're there to both learn and work. Okay. You want to get things done. Uh, I think probably the pioneer of, uh, the modern day dojo from a, from a tech standpoint was, uh, at target. So you can go to dojo, that's target.com. They're very upfront and open about how they run their dojo's. And it's a huge part of their culture right now. Uh, the other thing, and this is what we did when we got started, go and check out the book, uh, that was put out last year, uh, getting started with dojo's, um, Ross Clanton, Jackie, the Jew, the mine now, sorry for the mispronunciation, Carmen Deandra. Diardo John Sr and Eric Hosmer. Those were the authors of it. And they did a great job, really laying out what you need to do in order to, uh, get, uh, successfully get a dojo set up in your own environment.

00:09:23

So now what can you expect once you get a D dojo set up? Well, you bring your team and your own backlog. And that's what we mean. When we say we're there to work. You're not there to figure out exercise files and that you're going to throw away after you learn, you're bringing your real work to the table and you have experts in place that will help you get through the project and do it in a way that is product focused, as well as using good agile techniques and DevOps techniques. So what often happens is you create a six week a dojo experience with two and a half day micro sprints, uh, where you can fit 12 and through the entire life cycle of, of it. And you want to come out and you want to have a better understanding and grasp of agile and project.

00:10:09

We want to have a better, uh, DevOps and you want to be cloud native. Uh, when you do these things, you don't have to be, this is not a set in stone, but these are things that you can take while you have the time and have the people are dedicated to learn these things. You can take that time to actually make, uh, everyone a little bit stronger in their individual expertise. And then after six weeks, which basically you will have improved tech skills. And in our case, we were also looking to get faster time to value.

00:10:40

The one thing that we had to do, because this was not an official program funded by Comcast yet, or sky, nobody was going to let us take these people for six weeks and, uh, dedicated to it. So we are working to, uh, convince everyone that this should be a program that we accept. So we work more with one and two day sprints at this point, just to really get some value out of it so that we can then go back and say, why don't we go ahead and take a chance and make it a much longer, uh, timeline and really use the dojo the way it's supposed to be used. Okay. So when we started the dojo in this case with sky, uh, we, we, we went in with a plan and the plan when you're talking about dojo's is the working charter. So we had a working charter between sky and Comcast and the blank charter looks, something like this, and this is what we ended up with when we filled it out. So initially, do you want to kind of walk everybody through the parts of the charter so people know what it's all about?

00:11:42

Absolutely. So, uh, the important bit is obviously to name it, um, in this instance, we named it vinyl, DNS app guy for obvious reasons. Um, but it just gives you something to focus on. Um, and then you move on to the elevator pitch, um, which is, um, the idea would be, if someone walked into the room, they would know exactly what we're doing. So in this instance, if somebody joined a digital room, um, they could see what we were doing, what we were doing with our day. Um, and then they can walk straight out knowing that we were getting on with something, um, I'll leave software strategy elements to you, Mike.

00:12:18

Yeah. This one is, um, somewhat specific to Comcast. We have a group of, uh, 15 high level software strategy elements. So one of the ways that we want to be able to convince leadership that this is something we can do is in our charter. We want to basically mark down, which of the software strategy elements are we addressing when we come together and have this, this dojo? Um, we've redacted that for the purpose of that it's Comcast, uh, material, but this is a good open space to basically put maybe your software strategy, uh, elements there,

00:12:52

Goals and measures as it states, uh, you know, you have a goal and how is it measurable? Um, and again, Mike, uh, if you want to elaborate on that because you do it so well.

00:13:01

Yeah, I appreciate that. Uh, absolutely. So, you know, if you want to come in to the dojo with goals, right? And so many times we create goals and, um, I don't know if we make them measurable. It's very important in, in this case that you make the goals measurable. You want to have, um, some wildly important goals that you want to complete, and you want to have measures to make sure that, you know, that those things were complete, uh, teams work with KPIs, okay. Ours, things like that. It's basically the same idea, but here we have goals and measures.

00:13:31

So the skills matrix, it defines what technically, each person would be learning or in this instance, upskilling by the end of the dojo sprint. And one thing I love about the skills matrix setting out from the start is that you can see that everybody is going to learn something new. Um, so you, you feel confident regardless of what level you're at, that your you're not talking to somebody or dealing with somebody that has everything or all the knowledge that they need. Actually, everybody will learn something new from this.

00:14:00

I love it. I love that part as well.

00:14:03

And the working agreements, uh, for the DJ really important just to set your day out. Um, so you know exactly where everyone needs to be throughout the day. So you get the full day in. Um, of course everybody needs to have certain types of breaks in a normal working day. So you set the tone from the start, or are there any other meetings that other people need that, that people in the dojo need to attend? Uh, what breaks are you going to take for a, you know, a bit of rest, buy a coffee, break, a lunch break, um, and just letting everybody know where you're going to be at during the six, seven hours that you're doing your sprint, um, just to set it out for the day. So we know what to expect, basically.

00:14:37

Yeah. And of course, and this is super important when it's international and you have to work with those time differences, being able to really come together and agree on what time periods are going to overlap between the two groups. So this was very, very, okay. So when we came together, we were agile. So I'm going to go back to January. This was not when sky and Comcast came together. This was an internal project that we did, uh, at Comcast using the dojo model that was very successful. And, you know, we had a scrum master and we had, uh, physical scrum boards. That was something that we couldn't do. Uh, number one with the pandemic that came along. But, you know, also all of a sudden we're working with an international crew. So, you know, we, we needed something that would help us come together and collaborate, uh, even when we couldn't be co located. So what we used was a tool called mural, which I think we were very happy with. Uh, what did you think of the tool mural? Uh,

00:15:36

Mean, I loved it. It looked like the white boards that we would be standing next to you if we were in a physical room. Um, I love the fact that this, you would stick it on, like you're using post-it notes and, um, everybody had a, you know, a set of colors and it was very apparent who was writing what, um, I loved the fact that the sticky note, um, style of working prompted others in the, in the dojo to think about other aspects. So someone would write something on a post-it note and it would prompt it would provoke thought process. Um, but I also liked the fact that, um, there was so much transparency and seeing how two different companies think just laid out that way, where we really different or actually will be quite similar. So it was really good to have that layout.

00:16:21

Yeah, I agree. And one thing that we added on this, and I don't think I've ever used on a, on a physical scrum board before was we used these notes to also have heads up on when we're going to be those times where people had to go away from meetings or take breaks and things like that, the, the pink ones on there were there. And it was great to always kind of constantly be able to go back and say, oh, a break is coming up. You know, that reminder on there. So I, I really enjoyed, uh, the, the mural interface as well. Okay. So we were product focused. Um, that was something that we thought was really important. So I can tell you that, that one person, this was everybody that was there. The day of the dojo, one person in this picture is the product owner.

00:17:01

So try to figure out, sorry, the product manager try to figure out which one of these people you think is the product manager. And we'll see if you're right at the end of this short little game doing so first we know it's not Nisha, Nisha's the boss. He was not the product manager. Okay. This is Pete. Uh, the interesting story about Pete is, uh, his name is, is Peter last initial C. He has the same exact name as a person that was sat right next to him when he started working at Comcast. So imagine walking up to your cubicle and realizing that the person sitting right next to you has your same exact name. So Pete lovingly became PC too, but he's number one in our hearts, but he's not the product manager, Ryan, uh, we're all convinced he's a robot, right? Ryan speaks computers, you speak zeros and ones. He's amazing. He definitely was not the product manager. Uh, I'm not the product manager. Uh, I'm just happy that everybody let me stay in the room here. We were having so much fun and work needed to get done. And everybody kept reminding us that, uh, maybe Mike should stop making so many jokes.

00:18:09

It wasn't Dawn, but, uh, Don, uh, of course by the t-shirt you can tell is that sky over in London and we have healthy debates on what real football is. And I'm sure that she'll figure out what real football is before I'm done with our conversations. It was not Ali, but, but being able to see screen shares of Ali as he was, uh, handling DNS infrastructure and, uh, and, and Azure, uh, you can see that he had so much power as an individual over there. Uh, we loved watching the fact that we didn't have to go to all kinds of different people. Having Ali on the team allowed us to move so fast. It was great having him on the team. So it only leaves two people left and it is not. Should you, uh, should you was on the Comcast side, uh, AKA Tupperware, because he's always willing to sell you on, on containers. So that leaves Paul Paul was our product manager. If you guessed Paul, you were right. We made sure to have a product manager in the room. And that was super important to us because one of the cards that we put up had to do about with making sure that sky, uh, explained to us how DNS maintenance happened, uh, currently. So can you just explain a little bit what, uh, how it works currently, Nisha?

00:19:25

Yeah, sure. So it's exactly what Paul took us through. Um, we currently have a setup where, you know, raising a DNS request requires a certain levels of approval, uh, specifically three types of groups. Um, in our instance, we raise in our team, DNS entries often, and actually we are one of those groups that technically approves our own DNS changes. Um, so, or each other's DNS changes. So yeah, that's the current process at the moment, there is SLS involved and approval levels.

00:19:58

Yeah. That was really enlightening for us because, you know, we didn't want to just come in and say, we're just going to set up this application. We have no idea how you use DNS. It was great to get to know how you guys do it. And then also how we do it. And then of course we were DevOps. Um, and H how did that manifest itself? Uh, it was great to have people like, uh, like Dawn and Nisha in the room and, and Paul who, you know, are self-proclaimed non-technical people, uh, and then to have Ali be going onto Azure and creating a boon to instances and things like that, uh, to be able to give a little bit of insight of what the technology people do, uh, when they're getting stuff up and running for you, Nisha for your needs, uh, want to share anything about what it was like being, sitting there with all these?

00:20:49

Yeah. I mean, it was for someone that's, non-tech dealing with the developers that were on this dojo, it was, um, it was eyeopening. What struck us the most was the fact that you guys, as developers were extremely flexible, uh, accommodating you were, you were curious, I think that was really important. Um, you weren't just stuck in your own design. Um, you were interested in how another business actually does what they do in order to accommodate what it is we were trying to achieve here. Um, and so that was really nice for Dawn, myself and Paul, who are, non-developers working with developers in, in quite alien environment. And you didn't make us feel like we were working with aliens. So that was really cool for us.

00:21:31

Very cool. Very cool. Yeah. And so one thing, uh, that I would mention was even at times when there was like long, uh, uh, processing times that we had to wait five minutes for something to happen on the Azure side, uh, people were so willing to jump in and, uh, to, to follow up on what Nisha was saying. Uh, at one point we were talking about Dockers and containers and, uh, Nisha and Dawn didn't understand what we were talking about during a break. She just decided to share his screen and go over a little bit of a presentation of what a Docker and what containers are. So it's, it's the dojo allows for stuff like this, just ad hoc learning, um, to, to get everybody involved so that we can all make great products together.

00:22:17

And then this was the big moment. This was the moment when, uh, we actually saw our application running for the first time, uh, on not Comcast infrastructure. This was out there in Azure available and, uh, Ali brought it up and we all, you can see the adulation on everybody's face. Um, we were excited to have something out there so that Misha and Dawn and the others can actually explain to people what vinyl is and almost, uh, do a road show. So, so there, uh, any aspects about vinyl, DNS that kind of, uh, gets you excited to possibly work with Misha?

00:23:00

Yeah. I mean, it's such a great tool. I think the fact that, um, vinyl allows for these, um, this kind of see you can set the precedents with certain individuals to be able to go in and make DNS changes without the approval levels that we currently have. Um, you know, having availability to this code, even in a test environment to see it is so exciting. Um, being able to just make people feel empowered in their roles, uh, to say that they actually are, that they're trusted, they don't need to go in and have a DNS entry almost vetted, you know, and then approved and approved and approved. And then it goes ahead, you know? Um, so yeah, vinyl definitely offers, um, so much more flexibility and it, and it, and I've always the SLA is really, but like I said, it will really empower developers, um, to really see the benefits in what they're doing quicker. And I think everybody can combat for a developer wanting to get their work just done and out there. Right. So vital does that.

00:23:56

Absolutely. It definitely is important to note that once you get an infrastructure or a, uh, a footprint of DNS, that's a certain size, this idea of filling out tickets become so cumbersome. You know, you really need automation, uh, to just to give you an idea, we usually, uh, take on about 400 new zone creations per month, you know, and those are zones. Those aren't even the records that are within those zones, just new zone creations every month, 400, some of them get retired, but you know, that that's the kind of activity that you see in a, in something this size. So you even get the ability to go into your zone, see all the recent things that are, that have happened. Um, see the, the record set types that people are adding or taking away, uh, from, from DNS. It's very important to be able to go back for audit purposes and to see how people are touching your internal zones and records. So Nisha, now that you have an instance up and running, that you can get to, it's not behind our firewall, you're not blocked. Uh, how do you plan on taking that and, uh, using it to try to get DNS implemented fully at, uh, at sky?

00:25:11

Yeah, I mean, I absolutely love vinyl from the day you demoed it, uh, less than a year ago. Um, so, you know, for me, when I saw vinyl, I felt like it was very much Comcast. Um, and now just to see it set, sat on a sky owned domain, makes it so much more bespoke for us to go and show it. I think for me, it's about being a champion or something that I really think is of value to our business and let alone any business. So I think that the road ahead, it's a, it's about taking that, that test environment, um, and championing out to the right parts at sky, getting the right people connected with the tool, um, and really just highlighting the benefits for how it will empower us, you know, to do our work a lot more efficiently and, and drop a lot of administration, um, in our, in our own working environment, it's, uh, sky has great talent. So why do we need to add so many approval lines, you know? Um, and I think giving that talent, that ability to say, yes, we trust you enough to go in and make this DNS change without that many levels of approval will really change the dynamics of, I think the way people do think and do, do their, do, do their roles, if you get what I mean. So for me, the road ahead is championing at all to, to sky, um, and really getting it out there and getting people to get the feel of it.

00:26:35

I agree. We went through a major migration of vinyl DNS from, from a previous system here, and I got to see so many people see the power of vinyl DNS. And then once we got everybody migrated over, it's kind of business as usual at Comcast now. So I am super excited to see how, you know, once people started adopting it at sky at NBC universal and Dreamworks, uh, if possible, uh, I'm excited for that next level of excitement from the users. So, yeah,

00:27:03

I think one thing that's really good to know is that vinyl allows different types of, uh, levels and, you know, whether that's giving one person access to the root domain or sub domain, I think that's great. You know, you can set that bespoke approval level based on the type of team that you're dealing with.

00:27:20

Yeah, definitely. And so how can you help? Well, this is an open source project, so it's pretty easy. You can go to vital DNS, take a look at it, take a look at the, read me files, possibly bring it down for yourself. Point to another teammate who might be able to use it, give us a star. Those are all ways that you can help with that. Another way you can help is invite us back. You know, in, in another year, I would love to be able to say, you know, this was the beginning of our journey and, uh, perhaps be able to show a thriving, uh, group of people at a sky using vinyl DNS, uh, a year from now. Um, I'm hearing rumblings already that, uh, there's people that have started to take advantage over there. And we want to make sure that, that we're there to support them in any way they can with that. We thank you all for your time. Uh, we'll be ready to take any questions, uh, right after this and we appreciate you listening. Thanks very much.