Las Vegas 2020

Laying Down the Tracks for Technical Change at Comcast

In the 19th century, the big technology buzzword wasn't Kubernetes--it was railroads. Across the United States, regional railroads built by entrepreneurs sprung up around major cities. These were decoupled (literally!) from one another and often had incompatible tracks and equipment. When the railroads began to try to interconnect, it didn't run so smoothly at first.

If we fast forward to the modern day, many companies have seen a similar boom in promoting independent, Agile, DevOps teams. In a big enough company, "choose the right tool for the job" can often end up meaning "we've chosen every possible tool for the job" taken in aggregate. This leads to lost opportunities and friction when the technology organization is viewed as a whole and not solely through the lens of individual teams.


In this talk, we'll learn how the railroads eventually landed on standardized track widths and interconnectivity that led to greater overall commerce and prosperity. We'll draw parallels to initiatives we've undertaken at Comcast to drive more commonality in technology stacks to gain greater business leverage--initiatives you can bring back to try in your own companies.

JM

Jon Moore

Chief Software Architect & Senior Fellow, Comcast Cable

MW

Michael Winslow

Senior Director, Software Development & Engineering, Comcast

Transcript

00:00:07

Last year. I was so delighted that Jessica Sant and Ranga Mova, Vajra presented on the amazing 10 plus year DevOps journey at Comcast Jonathan Moore, their chief software architect and senior fellow was famously absent because of a famous ultimate Frisbee accident. Rumor has it, it began with him saying, let me show you how it's done, which then landed him in the hospital. So you may have noticed that the title of chief architect appears four times in the plenary keynote session talks this year. I am endlessly fascinated by this because I think the role of the chief architect can dictate so much of the structure of how teams work, hopefully for the better. And I'm so delighted that this year, John Moore will talk about how he views the role of architecture in large complex organizations and how it's manifested in the daily work of the 9,000 plus engineers inside of Comcast. I'm also incredibly excited that Jonathan will be presenting with another longtime friend within the DevOps enterprise community, Michael Winslow, who is just recently promoted to become a senior director of engineering. Michael is now leading the teams responsible for the technology behind the X one entertainment experience across mobile web and television. Here's Jonathan and Michael.

00:01:42

Well, Hey John, is that you? What are you doing there?

00:01:47

Oh, Hey Michael. Uh, well, I'm ripping up these railroad tracks.

00:01:52

You're ripping up railroad tracks. You might've asked why,

00:01:57

Uh, sure. You know how the railroad tracks that are coming into town, uh, on one side of town or a different width from the railroad tracks coming in on the other side of the town.

00:02:06

Sure do.

00:02:08

Well, the railroad companies are planning to relay those tracks so that the wits match up.

00:02:13

Yeah. I heard about that. I think it's great. Now people don't have to get off of one train lug all their baggage to another and get on another. They could just ride right through. It sounds like a workshop for everybody.

00:02:24

Well, it doesn't work out well for me. I own a hotel here in town and I sure liked to having people have to stop and stay overnight while they were changing trains.

00:02:34

Oh, well I guess change. Doesn't always work for everybody. Does it? You know, it's funny, I'm breaking character here, but this is a reminder of what we go through in the technology field when change is introduced. So, and that's pretty much what our talk here is about today. Uh, but John, one thing about this talk is I don't want you to be a executive the way that you are, right. I need you to go back and I need you to put your hat on and really treat things like an engineer.

00:03:04

If you say, so

00:03:07

Wait, I didn't mean an actual, you have an engineering yet. All right. Fine. Looks like it works for me. As a matter of fact, I'll do it with you. All right, let's do this.

00:03:21

Hi everyone. I'm John Moore, I'm chief software architect at Comcast cable. Um, and I do in fact have a, uh, an engineering hat and someday, if I got good enough, I would love to play electric bass and a rock and roll cover band.

00:03:33

And I'm Michael Winslow. I am senior director of Comcast and a fun fact about me is I still code to handle some of my management tasks.

00:03:43

All right.

00:03:45

All right. So this is us. This is Comcast NBC universal. Uh, we work for Comcast. And one thing about what we do and what we've been become over the past 10 years is we've gotten a lot larger with the acquisition of NBC universal and sky. We have so many operating companies and so many different technology stacks that we want to make sure that we as Comcast, uh, handle technology in a, in a, in a particular way, and that we share so that, uh, other companies that work with us, we can cross share how we do things and they can cross share with us how they do things.

00:04:25

All right, thanks for that overview, Mike. Uh, so as you might have, uh, surmised from the skit that we were talking about at the beginning, we're going to be talking about driving technical change across organizations. And in order to do that, uh, we're going to draw parallels with the way that the railroads developed in the United States, uh, through history. And so our story begins in 1834. And on this map, uh, we have highlighted in red, all of the railroads that have been completed at that point in time. Uh, as you can see, a lot of these are disconnected from one another, uh, literally decoupled, um, because they're point to point or, or regional networks. Um, and as a result, a lot of them were really locally customized. So some rail roads, for example, uh, decided to use a track widths or gauges that were four foot eight and a half inches wide.

00:05:16

That was a track age that was commonly used in Britain at the time. And that let them use. And by British locomotives, for example, uh, in New York, some railroads decided to go with a six foot wide gauge, even wider, uh, because they anticipated, they would be so successful that they were going to need really large engines to pull all of the cargo and passengers that they were gonna have on their trains. Um, this is a classic example of over-engineering, uh, meanwhile, down in the south, uh, a number of their railroads actually went with a five foot wide gauge, uh, because that was more convenient for shipping cotton bales from plantations, uh, out on the interior of the, uh, country out towards, uh, ports on the coast. And so, uh, this was the scenario here, um, at this point in time.

00:06:07

Yeah. And the thing that struck me when you first showed this map to me, and I saw all those red lines, so separated from each other, it reminded me of when we had all these different groups using all these different CIC tools, uh, here at Comcast. And just like you said, there were so many reasons to have these different gauges in these different rails for these, for these areas. Well, it's not like our teams didn't have good reason to use the tools that they were using. They all did, but it really was kind of a problem having so many different technologies under one roof.

00:06:38

Yeah. And I, in fact, the railroads ran into this once, uh, those railroads grew enough that they started needing to interconnect with one another. And inter-operate with one another. Um, just, just as a funny side note, uh, we, we actually didn't quite have enough CICT logos to cover all of the railroads. There's, there's one that we didn't manage to cover out in Western Tennessee. Um, but I'm not sure adopting yet another CICT system that it would have been a good reason to, uh, or that would have been a good reason for, for getting another one in here. So, uh, if we're going to try to consolidate these things, it's useful to know what we're going to try to consolidate on. And so if we go back to a little bit of the history of railroads, um, uh, it turns out that, uh, great Britain was sort of the Silicon valley of railroad technology, uh, at the time.

00:07:23

Um, they were really ahead in terms of the world in developing the technology, uh, of trans, um, and, and figuring out how to use them in 1845. Uh, there was a Royal commission, uh, that was charged with doing a study and making a recommendation on a railway gauges. Right. Um, and so just a reminder, again, a gauge of a railroad track is the distance between the two tracks. Um, and so they were asked to recommend a gauge to be used as a standard. Um, and sure enough, in 1846 here, here's actually a parliamentary commission that, that actually finished it study and came back with a recommendation and immediately got turned into a law. And, uh, and so here, what they did was they fixed, uh, the standard gauge in great Britain to four feet, eight and a half inches, which was something that was commonly in use at the time.

00:08:22

And what I saw when he showed me, this was the very next sentence. It says, oh, everybody, but Ireland, right. There was only one place that did not have to follow this rule right. As they were making this rule of standards. I thought that was pretty funny.

00:08:34

Yeah. I mean, it's a, it's really funny that, you know, on the one hand we're going to standardize and then actually as part of the same sentence, we carved out the first exception here. Now, you know, of course this makes sense, Ireland is its own island. I'm not sure that they necessarily have to connect their railroads, but, uh, but it was really very interesting. And so, uh, in essence, what we had was we had a group that did a study to try to recommend what the standard ought to be, um, and then resulting in an official decision as, as far as how that goes. Um, and at Comcast, we do something similar with our architecture Guild. So, um, so we have a, a grassroots organization called the architecture Guild and they produce architecture decision records. I think, uh, this is something that a number of companies use, but, uh, but this is a way of recording and making a broad technology decisions.

00:09:25

And so in addition to the decision and the rationale that comes along with making that decision, we also captured the context at the time, what were all the things that we knew or were considering when we were making this collective decision. Um, and then, you know, no decision, uh, you know, is completely side-effect free. Um, and they all come with trade-offs. And so we also, uh, try to record as many consequences as we can, uh, for, uh, for the decisions that we're making. And we, you know, we've ended up using this a number of times to talk about a number of different technologies over time.

00:10:02

Yeah. I remember specifically an ADR from the beginning. I believe the ADR started before I started, but it said that we needed to come up with a source code repository, uh, decision. And, you know, to me it was a no brainer, right. Get up, get hub, get hub. And I think to a lot of people, it was a no brainer. And don't, you know, one month later I'm working with a team who prefers Bitbucket, you know? So even when you think that something's a no-brainer with these ATRs, you have to kind of go through the process and realize that, uh, not everybody, not everything is common knowledge.

00:10:36

Right. So, uh, so we're going to walk you really quickly through a little bit of a use case. Um, so hearkening back to the situation that Mike identified, where we had a number of different D systems, um, I'm going to catch you up a little bit on, uh, how we push to CIC D decision through our architecture Guild. So, um, so every decision begins by identifying a working group, uh, to go and, uh, and do this work together. And it begins with a charter. And this really is a way just to declare things that are in and out of bounds. And what really are we talking about when we're trying to identify a capability that we want to make a common decision for? Um, as you may be able to see these are, these are stored in, uh, version control. Um, and we invite, um, pretty much anybody who cares, um, in the company to come and contribute to making the decision.

00:11:28

Um, and so that involves, uh, having people say, here's what I would like to use that for, um, bringing their use cases, um, identifying, Hey, if we're going to pick one that has to work for everybody, what are the properties that it has to have in order for that to be successful? Um, and then even collecting the list of things that we ought to consider, um, as we, as we go through trying to make a decision, um, and this is all crowdsourced and people can open pull requests against the ADR as it's being built up organically, um, in order to, to add their input and make their voice heard.

00:12:03

Yeah. And I think when John said anybody who cares can get involved, I think what he's really saying is even Michael can get involved. And because this was a, when we were coming up with the ADR for delivery, uh, we were using Jenkins on our team and I didn't know what the final outcome of the ADR would be. But what I did know was that we had very intricate, uh, static analysis and vulnerability scans in our pipelines. And I wanted to make sure that I can still pipe those same tools, use those same tools in whatever solution we went for. So I created an issue in the ADR that said, let's talk about static analysis and vulnerability scans. And, uh, so people can, can decide which of these solutions will actually be able to support what we had already.

00:12:49

Yeah. And it was great. And it was, you know, we definitely heard a lot from a lot of people, um, that, you know, everybody could bring all those use cases so that we could really make sure we consider them. Uh, some of you may notice that, that this sounds actually very similar to the process that the, the IATF uses, uh, to produce internet standards. Um, and RFCs, um, and in fact, uh, we w this is, uh, an actual explicit decision we made when we set up the architecture Guild. Um, and one of the things that the IATF talks about, um, is achieving rough consensus. And so I'm going to walk you through a little bit about how we drive this type of consensus. So, um, you know, one of the things that we don't do, uh, once we've identified the requirements and we've identified the set of potential solutions, we want to consider, we do not have an immediate popularity contest.

00:13:38

We do not ask for a vote, which one of these five things should we use. Um, instead we do something different, um, for each potential solution. Um, once we've done some research on them and share that research with everybody, we ask everybody to rate that individual solution on a scale of one to five. Um, some of you may recognize us as a fist of five technique. That's really common in some agile methodologies, but the idea is that, uh, I can look at a potential solution and I can rate it from five being the best solution ever down to one, which would be, this will be a terrible mistake. Um, but the interesting thing is that this gives us a lot richer data and shows us a histogram of support for each of these things. So, um, in particular, uh, I've shown the results of this poll for two different solutions here.

00:14:25

Um, and, and you're going to see some really interesting things, like, for example, uh, the number of people that rated each one of these, um, as the best solution ever is, is really far from a majority. Um, on the other hand, the set of people that rated it, um, as, at least acceptable was actually pretty big for both of these solutions, right? Almost 70% in both cases. Um, and so at this point, we knew that, um, Hey, these were really pretty likely that we could make this work for most people, but there still were these, uh, you know, the folks that rated at ones and twos. And so the next thing that we asked was we said, Hey, if you've got an issue, please raise it. Right. And so, uh, just as Mike D. Michael did before, um, you know, people would raise an issue and ask a question, and a lot of cases, uh, somebody who's used to doing something a one CSED system, and they don't know whether they can do it in a, in a different, uh, proposed system.

00:15:20

And so this gave an opportunity for people who were more familiar with that platform to come back and actually say, oh, you want to do that? Well, here's how you do that. Um, in, in this particular CICT system. And so what we found was that we were able to take some folks that, you know, had some concerns or issues and actually move them, uh, sort of a lot higher in terms of acceptance. And so, um, by pushing things through, we eventually landed on, uh, choosing Concourse, which is an open source, uh, CICB platform. We're going to mention it a little later. Um, but for that, that became our four feet, eight and a half inches standard gauge. Um, just like we saw it in the railroads after making that decision. Um, you know, one of the things we found was that, Hey, you know what, there were some issues that, uh, that we couldn't totally address. And we capture those in the consequences. You know, we said, you know what, it may not be totally convenient yet, uh, to solve these couple of things. And again, this was a way to make sure that those people that did have concerns, they could see that their input was reflected in the final document. Um, and that gave them a way to, to get bought into what we were doing. Um, but, but that's not the only place that we, you know, we we've used ADR is for more than just a CIC D

00:16:37

Yeah. As a matter of fact, to bring it a little bit more current, um, the fact that we use ATRs actually helped us in, uh, when something came about earlier in the year with social injustice, uh, we realized that, uh, actually John really spearheaded the idea that we wanted to adopt the inclusive naming standard that other people have been doing around around the industry. And so the good thing is at a company like this, sometimes the big question is how do you get started? What's the first step, but that is not something that we worry about. The first step is clearly known, create an ADR and invite people to come talk about it. And that's what happened here. And we're now in the stage of implementing, but we did put, we did go through quite a process in ADR process for the inclusive naming standards.

00:17:26

Yeah. And it was great to hear input from, uh, from everybody here. And, you know, I definitely found that I learned a lot in the process as we went. So, um, so that generally describes the way that we involve our practitioners in making the technical decisions, but, uh, at some point you actually need to, uh, make some changes happen. Um, and so let's back to the railroad. So in, uh, in 1862, uh, the Pacific railway act was passed, and this was the one that chartered the transcontinental railroad that would link the east and west coast of the United States. Um, and in a follow-on bill in 1863, um, Congress actually fixed, uh, the, the gauge of that big project, uh, to fix it at four feet, eight and a half inches. In other words, adopting the standard gauge, uh, from Britain. Um, and so, uh, you know, here was, here's a case where I was really the first time that the federal government had weighed in on, uh, the standardization of, uh, railroad track ages.

00:18:32

Um, and so, uh, you know, something similar, very similar happened with us, uh, inside Comcast. One of my favorite sayings is to never let a good crisis go to waste. Um, and as it turned out, as we were beginning to wrap up, uh, the ADR process for, uh, CIC platforms, um, it turned out that there was, uh, one particular week, um, after months of stability, where there were a number of incidents that, uh, that happened as a result of deployments. And, uh, what was determined there was that better deployment automation really was a way to help that, um, because enough of them have, and at the same time, this got the attention of senior leadership, and we were able to make the case that, Hey, you know what, um, if we actually adopted a common platform and we chartered a team to run that platform, um, then if we want to improve deployment automation, it's that much easier for other teams to copy the good practices, um, from, from each other. Um, and so, uh, we actually managed to get them to buy in, um, and even kicking headcount to create a team, uh, to go ahead and help us do this.

00:19:37

Now, would you say John, that this crisis helped everyone shoot, shoot, choose Concourse?

00:19:46

Uh, I like where your train of thought is going, but, um, but it, you know, I think one of the things that did happen was because we were very close to wrapping up the decision. It did encourage us to finish that process out. Um, so we could get going while we had the attention of senior leadership. Um, but, but maybe you can talk a little bit about how this manifested, um, as an engineering leader yourself.

00:20:09

Yeah. So, you know, we didn't see it coming with w we were a part of the ADR, but we didn't know we were that close to making a decision. Um, and the way we started seeing it on the front line was, uh, I remember vividly one specific, uh, cloud conference that was thrown by a completely different group than ours. And there, uh, as far as, as far as, uh, the key initiatives that Yon Hoffmeyer, who's our chief network ops officer, uh, said we were doing in his all hands, he decided to call out that we were going to adopt Concourse as part of, as part of this. And that was the beginning of me realizing, okay, this has buy-in from senior leadership, so we should probably start taking it seriously and possibly getting it into some of our okay. Ours.

00:20:58

Yeah. And this was, you know, this, this level of buy-in and support from senior leadership was really important. Um, you know, in addition to these types of messages that let everybody know this was coming, uh, we were able to lean on these senior leaders when we actually came to, to doing the engagements, um, and rely on them to, to continue to reinforce and echo, uh, the message down to their teams, uh, as we engage with them. But, uh, getting the buy-in of senior leadership is one thing, getting everybody else's, uh, buy-in as a, as a completely different, uh, sort of beast. And so, uh, I want to harken back to our little skit up at the beginning, uh, where I was ripping up the railroad tracks in Erie, Pennsylvania. Um, and so, uh, so let's, let's see what was going on there. So, uh, so here's a map that shows, uh, the, the Erie, Pennsylvania area, and as it turned out, there were three different railroad companies involved.

00:21:49

So the Buffalo on Stateline railroad, um, built one railroad that extended from Buffalo to down to the, uh, Pennsylvania New York state border. And they used a gauge of four feet, 10 inches. There was a second company, the area Northeast railroad that went from the New York, Pennsylvania border, uh, out to Erie. They use the gauge of six feet. Um, and then a third company, which was the Franklin canal company, uh, that went from Erie west to the Pennsylvania Ohio border. Um, and again at four feet, 10 inches. Um, and so the result was, if you were traveling on train from Buffalo, New York to Cleveland, Ohio, uh, you would actually have to change trains, uh, once, uh, at the Pennsylvania New York border, because the train that you're on, literally couldn't run on the next track. Um, and then you would have to do it again in Erie, Pennsylvania.

00:22:43

Um, and as a result of that, uh, there were, there were often a lot of stopovers, um, as cargo had to be loaded and unloaded, and passengers had to be loaded and unloaded as well. Um, and, uh, and so as it turned out, everyone in the town of Erie and a lot of business owners were actually really invested in the status quo. Um, we actually found a, an interesting, uh, we actually found the, the news article that actually talked about the events, uh, that, that we depicted there in our skit. Um, and so as the railroads, uh, companies, uh, began to standardize because, um, one of the, one of the three roads ended up buying the, the middle one with the sort of unusual gauge. And I said, oh, we're just going to streamline everything. Uh, the local residents began ripping up the news standardized tracks as they were coming, because they really didn't want it to go on.

00:23:36

Yeah, that's the first thing that I saw when you showed me this was, wow, they're calling people rowdies in the newspaper back then, you know, I don't even know if I'd feel bad if somebody, me a rowdy.

00:23:47

Yeah. Um, I, you know, hopefully we don't have teams that are getting into actual, uh, destruction as we're going, but, uh, but needless to say, you know, it's often the case that when we ask people to change, cause remember, think about all the logos we had of different CIPD systems up, uh, at the beginning. Um, no matter which one we pick, there's a lot of people that are going to have to change. Um, and so we, uh, adopted a framework that helped us, uh, think about this in a, in a structured way. Um, and so that framework is, is known as ADKAR, um, which is an acronym. Um, and the idea is that you want to walk people through a series of changes. Um, and so first, uh, you need to make everybody aware, uh, that, uh, that a change needs to happen. Um, and then you also want to build up a desire to see the results of that change among people.

00:24:38

And this, this really relies a lot on communications, such as, uh, for example, that executive who was giving a, a talk and highlighting it, um, in his opening address, um, after people want to have the change, you need to make sure that they know what the change entails, and they have the knowledge of, of what that, uh, what's going to be involved. Um, but they also need the ability to actually make the change. Um, and a lot of times this, this boils down to, uh, training and support for migration. Um, and then finally you want reinforcement to encourage people to make the change, but then also to make sure that the change sticks, and this is largely a function of, of leadership.

00:25:14

I remember getting so excited when John showed me that slide, that he was, that he was gonna use the ADKAR acronym and he would know about this, but we recently along with several other, uh, professionals, myself and, and, uh, some others you can see here, we created a book under it revolution that goes over the same concept pretty much, except for that, that, uh, vertical, uh, part of the axis that you see where they talk about, you know, having a, uh, knowing that change is necessary, that kind of maps well with the first aid and add car with the awareness and desire, but even with the awareness and desire, you're going to need that horizontal, uh, thing, which is your ability to change. You know, one thing is to know change changes needed. Another thing is to have the ability to make the change. And if you check out our book change, uh, in a successful organization, we kind of break that down into four quadrants based off of your level of ability and your level of desire. So highly encouraged to check that out.

00:26:16

Yeah. And so, you know, when it came to making this change for CSV systems, we, we intentionally, uh, did a lot of these things. And so we created a custom content, um, including very high quality tutorial videos, uh, that, uh, that went over, uh, the high level concepts, um, and, uh, and explained why we were doing some of these things, right. This is a, uh, slightly over five minute video that we use to introduce people to, to what going on here. Um, and then we, we also developed some internal training modules. And so, um, one of the things that we did was we divided the overall organization up into boarding groups, um, and as what we called them. And so we had a way to roll this out in a phased fashion. And so there was a specific engagement, the boarding process where teams would get the communication reinforced, and they would go through some specific training. One of the nice things about choosing an open source, um, platform like Concourse is that there's existing documentation and training material that exists. Um, and so, uh, so it was great. We could leverage that, but, but we were able to also layer on a few things that were specific to our implementation of it, um, at Comcast. And this was, um, we found this to be really successful in running hands-on workshops to help people develop the skills they needed to, to actually adopt a new platform.

00:27:39

Yeah. And I know at this conference, there's the concept of the experience report where, you know, year after year, uh, people are invited back to say, how are they progressing on the things that they had mentioned in the previous year? And what we noticed internally is we have so many opportunities to do that from one internal conference to the next or all hands meetings, or, uh, you know, uh, all hands meetings. So basically we are able to see when other teams are making progress on the ADR. And so when our team started seeing that, you know, another team had already got done moving all of their people over to Concourse that gave us the idea that we, we really better get started on this. And I think that was a good mechanism to make sure people would, uh, keep moving forward. And so when you had the idea in the beginning that senior leadership is, uh, supportive of it, and now you have this sort of friendly competition where you're seeing other teams implementing it at the same time, and you don't want to be left behind or last,

00:28:44

Um, Mike, maybe you can talk a little bit about, you know, what was involved with, you know, your team actually making the change when say once they decided to, to get going on it.

00:28:54

Yeah. So for us, it was basically, we had to take a look at all of our, uh, Jenkins, uh, pipelines, all the groovy code that we had, whether or not it was, uh, in the application itself or separated and, and kept in source control and figuring out how does that map over to a Concourse pipeline. And there are a lot of, uh, differences and it was not easy. It was very, very hard in the beginning, but the work paid off in the end, just knowing that this is kind of off of our plate now, you know, we do, we still do, uh, our own pipelines, but the, uh, the managing of it, the making sure that it's, uh, up all the time is a shared service. And we appreciate being able to, uh, concentrate on what we do best, which is create software. And then the, uh, CIC D team, the delivery team gets to concentrate on keeping our pipelines up and running.

00:29:47

Yeah. And I, you know, and this is actually one of the things that that's worth noting here, right. Is that your, your team didn't actually, uh, get any short-term value out of this. Right. Um, you know, before you're on Concourse, uh, you were doing some set of staff with TICD and you were doing the exact same set of stuff once you had moved over. Um, so, so that, you know, it's a lot of short-term work for no apparent short-term gain. Um,

00:30:15

Yeah,

00:30:16

Exactly. Um, now one of the things that we did end up finding though, was that I'm in the process of doing this, we're able to build a really large internal community, um, around, uh, Concourse. And, you know, in fact, uh, the last we checked, it was up over 2000 people. Um, and this, this considerably made it easier for teams to adopt it because when, uh, teams were doing the type of analysis that Mike was describing of, okay, how do I do X in Concourse? Um, they could just come in to this channel and ask and sure enough chances were that somebody else has already done that. And then they can actually just send you a link, uh, to their pipeline configuration, um, that, that actually illustrates it an action. And so, so we really did find that building up a giant community inside what really made this a lot easier for a lot of teams to, to make a transition.

00:31:04

Um, and you know, of course the, you know, there is, uh, some interesting, you know, real results here. So, um, so some teams have really taken their CICB practices, uh, all the way. Um, I think, uh, in a really interesting direction. So there's an actual button, um, that if you push it, it will attempt to take whatever is checked into Maine and put it in production. Um, and this team had developed such confidence in their CIC pipeline, um, that they knew that if you punch this push button, that their test automation, their deployment automation, uh, was, uh, something had so much confidence in that, that pushing this button at any time, uh, was going to be something that was safe to do. Um, as it turns out, this is also one of the teams that was an early adopter of Concourse. And so, um, because they had implemented a lot of this, uh, advanced deployment automation, um, and test automation, um, in Concourse, it made it that much easier for teams to adopt those same practices at the same time that they were migrating to Concourse. Um, and so we really have started to see this really pay off for us.

00:32:10

Yeah. I remember that button. I pushed it once they let me push at one time. So, so what are we looking for? Well, one thing is, um, we want to have a conversation, especially if you're trying to introduce the same kind of commonality across your enterprise, where, uh, you have a lot of independent and self-empowered teams.

00:32:29

Yeah. We're, we're really curious if you're working on this, uh, what has worked for you, um, and what have you tried maybe that hasn't worked? Um, we would love to swap stories with you and, and collaborate on this because, uh, this is an important concept and a, and a problem that we think is, uh, is pretty common across our industry. Um, and I think we can solve this better, uh, by working together on it. But in the meantime, uh, you know, let's summarize some of the things that we've learned while we're trying to do this. Um, uh, I think we'll start with the observation that Mike made at the beginning, uh, where, um, if you do empower teams to choose the right tool for the job, um, but you have a large enterprise that can result in that you've chosen every tool for every job, uh, at the same time.

00:33:14

Yeah. And one thing that we saw, uh, is that the more freedom you give, the more variety that you have sometimes that causes friction, uh, it also prevents maybe some of your individual contributors to be able to move from team to team and take their knowledge with them. Uh, and it also can cause some issues with calibrating across teams. Another thing is, uh, again, hearkening back to the beginning, just because we can see some people can see the global benefits of standardization. It's not always that clear locally to everybody. Not everybody feels as if these benefits are going to, uh, uh, really be beneficial to them.

00:33:54

Yeah. And, uh, you know, fortunately one of the things that we have found is that, uh, we found that taking a structured approach can work for these types of changes. So, uh, whether that is, uh, having a structured way for practitioners to participate in technical decisions like our, um, architecture Guild that, that collaboratively generates the architecture decision records, um, you know, making sure that we take a structured approach to getting buy-in at all levels of the organization from senior leadership to middle management, to the frontline practitioners, um, and then actually taking a structured approach to, to provide training and support, uh, to make sure that the change is successful, um, as you do it. Um, and so, so that's what we've learned. Um, hopefully you can take some of this back to your own companies, um, and maybe, uh, that will help your own transformation efforts, uh, pick up steam. Thanks.

00:34:46

Thanks everyone.