Las Vegas 2020
Laying Down the Tracks for Technical Change at Comcast
In the 19th century, the big technology buzzword wasn't Kubernetes--it was railroads. Across the United States, regional railroads built by entrepreneurs sprung up around major cities. These were decoupled (literally!) from one another and often had incompatible tracks and equipment. When the railroads began to try to interconnect, it didn't run so smoothly at first.
If we fast forward to the modern day, many companies have seen a similar boom in promoting independent, Agile, DevOps teams. In a big enough company, "choose the right tool for the job" can often end up meaning "we've chosen every possible tool for the job" taken in aggregate. This leads to lost opportunities and friction when the technology organization is viewed as a whole and not solely through the lens of individual teams.
In this talk, we'll learn how the railroads eventually landed on standardized track widths and interconnectivity that led to greater overall commerce and prosperity. We'll draw parallels to initiatives we've undertaken at Comcast to drive more commonality in technology stacks to gain greater business leverage--initiatives you can bring back to try in your own companies.
JM
Jon Moore
Chief Software Architect & Senior Fellow, Comcast Cable
MW
Michael Winslow
Senior Director, Software Development & Engineering, Comcast
Transcript
<silence>
Last year, I was so delighted that Jessica San and Ranga Mova RA, presented on the amazing 10 plus year DevOps journey at Comcast. Jonathan Moore, their chief software architect and senior fellow, was famously absent because of a famous ultimate Frisbee accident. Rumor has it, it began with him saying, let me show you how it's done, which then landed him in the hospital <laugh>. So you may have noticed that the title of Chief architect appears four times in the plenary keynote session talks this year. I am endlessly fascinated by this because I think the role of the chief architect can dictate so much of the structure of how teams work, hopefully for the better. And I'm so delighted that this year, John Moore will talk about how he views the role of architecture in large, complex organizations and how it's manifested in the daily work of the 9,000 plus engineers inside of Comcast, I'm also incredibly excited that Jonathan will be presenting with another longtime friend within the DevOps enterprise community. Michael Winslow, who is just recently promoted to become a senior director of engineering. Michael is now leading the teams responsible for the technology behind the X one entertainment experience across mobile web and television. Here's Jonathan and Michael.
Uh, Hey, John. Is that you? What are you doing there?
Oh, hey, Michael. Uh, well, I'm, I'm ripping up these railroad tracks.
You are ripping up railroad tracks. You mind if I asked you why?
Uh, sure. You know how the railroad tracks that are coming into town, uh, on one side of town are a different width from the railroad tracks coming in on the other side of town?
Sure do.
Well, the railroad companies are planning to relay those tracks so that the widths match up.
Yeah, I heard about that. I think it's great. Now, people don't have to get off of one train lug all their baggage to another and get on another, they could just ride right through. Sounds like it works out for everybody.
Well, it doesn't work out well for me. I own a hotel here in town, and I sure liked having people have to stop and stay overnight while they were changing trains.
Oh, well, I guess change doesn't always work for everybody, does it? You know, it's funny, I'm breaking character here, but this is a reminder of what we go through in the technology field when change is introduced. So, and that's pretty much what our talk here is about today. Uh, but John, one thing about this talk is I don't want you to be, uh, executive the way that you are. Right. I need you to go back and I need you to put your engineering hat on and really treat things like an engineer.
Well, if you say so,
Wait, I didn't mean an actual, you have an engineering hat. All right, fine. Yeah, totally. Looks like it works for me. As a matter of fact, I'll do it with you. All right, let's do this.
Hi everyone. I'm John Moore. I'm Chief Software architect at Comcast Cable. Um, and I do, in fact have a, uh, an engineering hat. And someday, if I get good enough, I would love to play electric base in a rock and roll cover band.
And I'm Michael Winslow. I am senior director at Comcast. And the fun fact about me is I still code to handle some of my management tasks.
All right.
Alright, so this is us. This is Comcast, uh, NBC Universal. Uh, we work for Comcast. And one thing about what we do and what we've been become over the past 10 years is we've gotten a lot larger with the acquisition of NBCUniversal and Sky. We have so many operating companies and so many different technology stacks that we wanna make sure that we as Comcast, uh, handle technology in a, in a, in a particular way, and that we share so that, uh, other companies that work with us, we can cross share to how we do things, and they can cross share with us how they do things.
All right. Thanks for that overview, Mike. Uh, so as you might have, uh, surmised from the skit that we were talking at the beginning, uh, we're gonna be talking about, uh, driving technical change across organizations. And in order to do that, uh, we're gonna draw parallels with the way that the railroads developed in the United States, uh, through history. And so our story begins in 1834, and on this map, uh, we have highlighted in red all of the railroads that have been completed at that point in time. Uh, as you can see, a lot of these are disconnected from one another, uh, literally decoupled, um, because they're point to point or, or regional networks. Um, and as a result, a lot of them were really locally customized. So some railroads, for example, uh, decided to use, uh, track widths or gauges that were four foot eight and a half inches wide.
That was, uh, a track gauge that was commonly used in Britain at the time, and that let them use and buy British locomotives, for example, uh, in New York, some railroads decided to go with a six foot wide gauge, even wider, uh, because they anticipated they would be so successful that they were gonna need really large engines to pull all of the cargo and passengers that they were gonna have on their trains. Um, this is a classic example of over engineering. Uh, meanwhile down in the South, uh, a number of their railroads actually went with a five foot wide gauge, uh, because that was more convenient for shipping cotton bales from plantations, uh, out on the interior of the, uh, country out towards, uh, ports on the coast. And so, uh, this was the, the scenario here, um, at, at, at this point in time.
Yeah. And the thing that struck me when you first showed this map to me, and I saw all those red lines so separated from each other, it reminded me of when we had all these different groups using all these different CICD tools, uh, here at Comcast. And just like you said, there were so many reasons to have these different gauges in these different rails for these d for these areas. Well, it's not like our teams didn't have good reason to use the tools that they were using. They all did. But it really was kind of a problem having so many different technologies under one roof.
Yeah. And in fact, the, the railroads ran into this once, uh, those railroads grew enough that they started needing to interconnect with one another and interoperate with one another. Um, just, just as a funny side note, uh, we, we actually didn't quite have enough CICD logos to cover all of the railroads. There's, there's one that we didn't manage to cover out in western Tennessee. Um, but I'm not sure adopting yet another CICD system would, would've been a good reason to, uh, or that would've been a good reason for, for getting another one in here. So, uh, if we're gonna try to consolidate these things, it's useful to know what we're gonna try to consolidate on. And so, if we go back to a little bit of the history of railroads, um, uh, it turns out that, uh, great Britain was sort of the Silicon Valley of railroad technology, uh, at the time.
Um, they were really ahead in terms of the world in developing the technology, uh, of trains, um, and, and figuring out how to use them. In 1845, uh, there was a royal commission, uh, that was charged with doing a study and making a recommendation on, uh, railway gauges. Right? Um, and so just, uh, a reminder again, uh, a gauge of a railroad track is the distance between the two tracks. Um, and so they were asked to recommend, uh, a gauge to be used as a standard. Um, and sure enough, in 1846, uh, here, here's actually a parliamentary commission that, that actually finished its study and came back with a recommendation and immediately got turned, uh, into a law. And, uh, and so here what they did was they fixed, uh, the standard gauge in Great Britain to four feet, eight and a half inches. Um, which was something that was commonly in use at the time.
Yeah. And what I saw when you showed me this was the very next sentence, it says, oh, everybody but Ireland, right? There was only one place that did not have to follow this rule, right? As they were making this rule of standards. I thought that was pretty funny.
Yeah, I mean, it's, uh, it's really funny that, you know, in, on the one hand, we're gonna standardize. And then actually, as part of the same sentence, we carved out the first exception here. Now, you know, of course this makes sense. Ireland is its own island. I'm not sure that they necessarily have to connect their railroads, but, uh, but it was really very interesting. And so, uh, in essence, what we had here was we had a group that did a study to try to recommend what the, uh, standard ought to be, um, and then resulting in an official decision as, as far as how that goes. Um, and at Comcast, we do something similar with our Architecture guild. So, um, so we have a, uh, a grassroots organization called the Architecture Guild, and they produce architecture decision records. I think, uh, this is something that a number of companies use, but, uh, but this is a way of recording and making, uh, broad technology decisions.
And so in addition to the decision and the rationale that comes along with making that decision, we also capture the context at the time, what were all the things that we knew or were considering when we were making this collective decision. Um, and then, you know, no decision, uh, you know, is completely side effect free. Um, and they all come with trade-offs. And so we also, uh, try to record as many consequences as we can, uh, for, uh, for the decisions that we're making. And we, you know, we've ended up using this, uh, a number of times to talk about a number of different technologies over time.
Yeah. I, I remember specifically an A DR from the beginning. I, I believe the A DR started before I started, but it, it said that we needed to come up with a source code repository, uh, decision. And, you know, to me it was a no-brainer, right? GitHub, GitHub, GitHub. And I think to a lot of people, it was a no-brainer. And don't, you know, one month later, I'm working with a team who prefers Bitbucket, you know? So even when you think that something's a no brainer with these ADRs, you have to kind of go through the process and realize that, uh, not everybody, not everything is common knowledge,
Right? So, uh, so we're gonna walk you really quickly through a little bit of a use case. Um, so hearkening back to the situation that Mike identified where we had a number of different CICD systems. Um, I, I'm gonna catch you up a little bit on, uh, how we push the CICD decision through our Architecture Guild. So, um, so every decision begins by identifying a working group, uh, to go and, uh, and do this, uh, work together. And it begins with a charter. And this really is a way just to declare things that are in and out of bounds. And what really are we talking about when we're trying to identify a capability that we wanna make a common decision for? Um, as you may be able to see, these are, these are stored in, uh, version control. Um, and we invite, um, pretty much anybody who cares, um, in the company to come and contribute to making the decision.
Um, and so that involves, uh, having people say, here's what I would like to use that for. Um, bringing their use cases, um, identifying, Hey, if we're gonna pick one that has to work for everybody, what are the properties that it has to have in order for that to be successful? Um, and then even collecting the list of things that we ought to consider, um, as we, as we go through trying to make a decision. Um, and this is all crowdsourced, and people, people can open pull requests against, uh, the A DR as it's being built up organically, um, in order to, to add their input and make their voice heard.
Yeah. And I think when John said, anybody who cares can get involved, I think what he was really saying is, even Michael can get involved because <laugh>, this was, uh, when we were coming up with the A DR for delivery, uh, we were using Jenkins on our team, and I didn't know what the final outcome of the A DR would be, but what I did know was that we had very intricate, uh, static analysis and vulnerability scans in our pipelines, and I wanted to make sure that I can still pipe those same tools, use those same tools in whatever solution we, we went for. So I created an issue in the A DR that said, let's talk about static analysis and vulnerability scans, and, uh, so people can, can decide which of these solutions we'll actually be able to support what we had already.
Yeah. And it was great. And it was, you know, we definitely heard a lot from a lot of people, um, that, you know, everybody could bring all those use cases so that we could really make sure we consider them. Uh, some of you may notice that, that this sounds, uh, actually very similar to the process that the, the IETF uses, uh, to produce internet standards, um, and RFCs. Um, and in fact, uh, we, this is a, an actual explicit decision we made when we set up the Architecture Guild. Um, and one of the things that the IATF talks about, um, is achieving rough consensus. And so, I, I'm gonna walk you through a little bit about how we drive this type of consensus. So, um, you know, one of the things that we don't do, uh, once we've identified the requirements and we've identified the set of potential solutions we want to consider, we do not have an immediate popularity contest.
We do not ask for a vote, which one of these five things should we use? Um, instead we do something different, um, for each potential solution. Um, once we've, you know, done some research on 'em and shared that research with everybody, we ask everybody to rate that individual solution on a scale of one to five. Um, some of you may recognize us as a fist to five technique that's really common in, uh, some agile methodologies, but the idea is that, uh, I can look at a potential solution, and I can rate it from five being the best solution ever down to one, which would be, this would be a terrible mistake. Um, but the interesting thing is that this gives us, uh, a lot richer data and shows us a histogram of support for each of these things. So, um, in particular, I, I've shown the, the results of this poll for two different solutions here.
Um, and, and you begin to see some really interesting things. Like, for example, uh, the number of people that rated each one of these, um, as the best solution ever is, is really far from a majority. Um, on the other hand, the set of people that rated it, um, as at least acceptable was actually pretty big for both of these solutions, right? Almost 70% in both cases. Um, and so at this point, we knew that, um, hey, these were really pretty likely that we could make this work for most people, but there still were these, uh, you know, the folks that rated it ones and twos. And so the next thing that we asked was, we said, Hey, if you've got an issue, please raise it. Right? And so, uh, just as Mike did, Michael did before, um, you know, people would raise an issue and ask a question.
In a lot of cases, uh, somebody's used to doing something in one CI ICD system, and they don't know whether they can do it in, uh, in a different, uh, proposed system. And so this gave an opportunity for people who were more familiar with that platform to come back and actually say, oh, you want to do that? Well, here's how you do that, um, in, in this particular CI CD system. And, and so what we found was that we were able to take some folks that, you know, had some concerns or issues and actually move them, uh, sort of a lot higher in terms of acceptance. And so, um, by pushing things through, we eventually landed on, uh, choosing Concourse, which is an open source, uh, CICD platform. We're, we're gonna mention it a little later. Um, but for that, that became our four feet, eight and a half inches standard gauge.
Um, just like we saw in the railroads after making that decision, um, you know, one of the things we found was that, hey, you know what? There were some issues that, uh, that we couldn't totally address, and we capture those in the consequences. You know, we said, you know what? It may not be totally convenient yet, uh, to solve these couple of things. And again, this was a way to make sure that those people that did have concerns, they could see that their input was reflected in the final document. Um, and that gave them a way to, to get bought into what we were doing. Um, but, but that's not the only place that we, you know, we, we've used ADRs for more than just, uh, CICD.
Yeah. As a matter of fact, to bring it a little bit more current, um, the fact that we use ADRs actually helped us in, uh, when something came about earlier in the year with social injustice, uh, we realized that, uh, actually John really spearheaded the idea that we wanted to adopt the inclusive naming standard that, that other people have been doing around, around the industry. And so, the good thing is, at a company like this, sometimes the big question is, how do you get started? What's the first step? But that is not something that we worry about. The first step is clearly known, create an A DR and invite people to come talk about it. And that's what happened here. And, and we're now in the stage of implementing. But we did put, we did go through quite a process an A DR process for the inclusive naming standards.
Yeah. And it was great to hear input from, uh, from everybody here. And, you know, I definitely found that I learned a lot in the process as we went. So, um, so that generally describes the way that we, you know, involve our practitioners in making the technical decisions. But, uh, at some point, you actually need to, uh, make some changes happen. Um, and so let's flip back to the railroad. So in, uh, in 1862, uh, the Pacific Railway Act was passed, and this was the one that chartered the Transcontinental Railroad that would link the east and west coast of the United States. Um, and in a follow-on bill in 1863, um, Congress actually fixed, uh, the, the gauge of that big project, uh, to fix it at four feet, eight and a half inches, in other words, adopting the, the standard gauge, uh, from Britain. Um, and so, uh, you know, here was, here was a case where, uh, it was really the first time that the federal government had weighed in on, uh, the standardization of, uh, railroad track gauges.
Um, and so, uh, you know, something similar, very similar happened with us, uh, inside Comcast. One of my favorite sayings is to never let a good crisis go to waste. Um, and as it turned out, as we were beginning to wrap up, uh, the a DR process for, uh, CICD platforms, um, it turned out that there was, uh, one particular week, um, after months of stability where there were a number of incidents that, uh, that happened as a result of deployments. And, uh, what was determined there was that better deployment automation really was a way to help that. Um, because enough of them happened at the same time, this got the attention of senior leadership, and we were able to make the case that, Hey, you know what, um, if we actually adopted a common platform and we chartered a team to run that platform, um, then if we wanna improve deployment automation, it's that much easier for other teams to copy the good practices, um, from, from each other. Um, and so, uh, we actually managed to get them to buy in, um, and even kick in headcount to create a team, uh, to go ahead and, and help us do this.
Now, would you say, John, that this crisis helped everyone? Chew, chew, choose concourse, get it?
Uh, I like where your train of thought is going, but, uh, okay. But it, you know, I think one of the things that did happen was because we were very close to wrapping up the decision, it did encourage us to, to finish that process out, um, so we could get going while we had the attention of senior leadership. Um, but, but maybe you can talk a little bit about how this manifested, um, as an engineering leader yourself.
Yeah. So, you know, we didn't see it coming. We, we, we were a part of the A DR, but we didn't know we were that close to making a decision. Um, and the way we started seeing it on the frontline was, uh, I remember vividly one specific, uh, cloud conference that was thrown by a, a completely different group than ours. And there, uh, as far as, as far as, uh, the key initiatives that John Hoffmeyer, who's our chief network officer, uh, said we were doing in his all hands, he decided to call out that we were going to adopt concourse as part of, as part of this. And that was the beginning of me realizing, okay, this has buy-in from senior leadership, so we should probably start taking it seriously and possibly getting it into some of our OKRs.
Yeah. And this is, you know, this, this level of buy-in and support from senior leadership was really important. Um, you know, in addition to these types of messages that let everybody know this was coming, uh, we were able to lean on these senior leaders when we actually came to, to doing the engagements, um, and rely on them to, to continue to reinforce and echo, uh, the message down to their teams, uh, as we engage with them. But, uh, getting the buy-in of senior leadership is one thing. Getting everybody else's, uh, buy-in is a, is a completely different, uh, sort of beast. And so, uh, I wanna harken back to our little skit up at the beginning, uh, where I was ripping up the railroad tracks in Erie, Pennsylvania. Um, and so, uh, so let's, let's see what was going on there. So, uh, so here's a map that that shows, uh, the, the Erie Pennsylvania area.
And as it turned out, there were three different railroad companies involved. So the Buffalo and Stateline Railroad, um, built one railroad that extended from Buffalo to down to the, uh, Pennsylvania, New York state border. And they used a gauge of four feet, 10 inches. There was a second company, the Erie Northeast Railroad, that, uh, went from the New York Pennsylvania border, uh, out to Erie. They used a gauge of six feet, um, and then a third company, which was the Franklin Canal Company, uh, that went from Erie West to the Pennsylvania, Ohio border. Um, and again, at four feet, 10 inches. Um, and so the result was if you were traveling on train from Buffalo, New York to Cleveland, Ohio, uh, you would actually have to change trains, uh, once, uh, at the Pennsylvania New York border because the train that you're on literally couldn't run on the next track.
Um, and then you would have to do it again in Erie, Pennsylvania. Um, and as a result of that, uh, there were, there were often a lot of stopovers, um, as cargo had to be loaded and unloaded, and passengers had to be loaded and unloaded as well. Um, and, uh, and so as it turned out, everyone in the town of Erie and a lot of business owners were actually really invested in the status quo. Um, we actually found a, an interesting, uh, we actually found the, the news article that actually talked about the events, uh, that, that we depicted there in our skit. Um, and so as the railroads, uh, companies, uh, began to standardize because, um, one of the, one of the three roads ended up buying the, the middle one with the, the sort of unusual gauge, and they said, ah, we're just gonna streamline everything. Uh, the local residents began ripping up the new standardized tracks as they were coming, um, because they really didn't want it to go on.
Yeah, that's what I, the first thing that I saw when you showed me this was, wow, they're calling people rowdy's in the newspaper back then. You know, I don't even know if I'd feel bad if somebody called me a rowdy <laugh>.
Yeah. Um, you know, hopefully we don't have teams that are getting into actual, uh, destruction as we're going. But, uh, be needless to say, you know, it's often the case that when we ask people to change, 'cause remember, think about all the logos we had of different CICD systems up, uh, at the beginning. Um, no matter which one we pick, there's a lot of people that are gonna have to change. Um, and so we, uh, adopted a framework that helped us, uh, think about this in a, in a structured way. Um, and so that framework is, is known as acar, um, which is an acronym. Um, and the idea is that you wanna walk people through, uh, a series of changes. Um, and so first, uh, you need to make everybody aware, uh, that a, that a change needs to happen. Um, and then you also want to build up a desire to see the results of that change among people.
And this, this really relies a lot on communication, such as, uh, for example, that executive who was giving a a a talk and highlighting it, um, in his opening address, um, after people want to have the change, you need to make sure that they know what the change entails, and they have the knowledge of, of what that, uh, what's going to be involved. Um, but they also need the ability to actually make the change. Um, and a lot of times this, this boils down to, uh, training and support for a migration. Um, and then finally, you want reinforcement to encourage people to make the change, but then also to make sure that the change sticks. And this is largely a function of, of leadership.
I remember getting so excited when John showed me that slide that he, that he was gonna use the ad car acronym, and he wouldn't know about this. But we recently, along with several other, uh, professionals myself and, and, uh, some others you can see here, we created a, uh, book under IT revolution that goes over the same concept pretty much except for that, that, uh, vertical, uh, part of the axis that you see where they talk about, you know, having a, uh, knowing that change is necessary, that kind of maps well with the first ad and add car with awareness and desire, but even with the awareness and desire, you're gonna need that horizontal, uh, thing, which is your ability to change. You know, one thing is to know change is needed. Another thing is to have the ability to make the change. And if you check out our book change, uh, in a successful organization, we kind of break that down into four quadrants based off of your level of ability and your level of desire. So highly encouraged to check that out.
Yeah. And so, you know, when it came to, uh, making this change for CICV systems, we, we intentionally, uh, did a lot of these things. And so we created custom content, um, including very high quality tutorial videos, uh, that, uh, that went over, uh, the high level concepts, um, and, uh, and explained why we were doing some of these things, right? This is a, a slightly over five minute video that, that we use to introduce people to, to what was going on here. Um, and then we, we also developed some internal training modules. And so, um, one of the things that we did was we divided the overall organization up into boarding groups, um, and is what we called them. And so we had, uh, a way to roll this out in a phased fashion. And so there was a specific engagement, the boarding process, where teams would get the communication reinforced and they would go through some specific training. One of the nice things about choosing an open source, um, platform like Concourse, is that there's existing, uh, documentation and training material that exists. Um, and so, uh, so it was great we could leverage that, but, but we were able to also layer on a few things that were specific to our implementation of it, um, at Comcast. And, and this was, um, we found this to be really successful in running hands-on workshops to help people develop the skills they needed to, to actually adopt the new platform.
Yeah. And I, I know at this conference, there's the concept of the experience report where, you know, year after year, uh, people are invited back to say, how are they progressing on the, uh, things that they had mentioned in the previous year. And what we noticed internally is we have so many opportunities to do that from one internal conference to the next, or all hands meetings or, uh, you know, uh, all hands meetings, <laugh>. So basically we are able to see when other teams are making progress on the A DR. And so when our team started seeing that, you know, another team had already got done moving all of their people over to Concourse, that gave us the idea that we, we really better get started on this. And I think that was a good mechanism to make sure people would, uh, keep moving forward. And so you had the idea in the beginning that senior leadership is, uh, supportive of it, and now you have this sort of friendly competition where you're seeing other teams implementing it at the same time, and you don't wanna be left behind. Or last,
Um, Mike, maybe you can talk a little bit about, you know, what was involved with, you know, your team actually making the change once they, once they decided to, to get going on it?
Yeah, so for us, it was basically we had to take a look at all of our, uh, Jenkins, uh, pipelines, all the groovy code that we had, whether or not it was, uh, in the application itself, or separated and, and kept in source control, and figuring out how does that map over to a concourse pipeline. And there are a lot of, uh, differences. And it was not easy. It was very, very hard in the beginning, but the, the work paid off in the end, just knowing that this is kind of off of our plate now, you know, we do, we still do, uh, our own pipelines, but the, uh, the managing of it, the making sure that it's, uh, up all the time is a shared service. And we appreciate being able to, uh, concentrate on what we do best, best, which is create software. And then the, uh, CICD team, the delivery team gets to concentrate on keeping our pipelines up and running.
Yeah. And I, you know, and this is actually, you know, one of the things that that's worth noting here, right? Is that your, your team didn't actually, uh, get any short-term value out of this. Right? Um, you know, before you were on concourse, uh, you were doing some set of stuff with CICD and you were doing the exact same set of stuff once you had moved over. Um, so, so that, you know, it's a lot of short-term work for no apparent short-term gain. Yeah. Um, but
A lot of work to achieve parity. Yeah,
That's right. Exactly. Um, now one of the things that we did end up finding though, was that, um, in the process of doing this, we were able to build a really large internal community, um, around, uh, concourse. And, you know, in fact, uh, the last we checked it was up over 2000 people. Um, and this, this considerably made it easier for teams to adopt it because when, uh, teams were doing the type of analysis that Mike was describing of, okay, how do I do X in concourse? Um, they could just come into this channel and ask, and sure enough, chances were that somebody else has already done that and then act, they can actually just send you a link, uh, to their pipeline configuration, um, that, that actually illustrates it in action. And so, so we really did find that building up a giant community inside what really made this a lot easier for a lot of teams to, to make a transition.
Um, and, you know, of course the, you know, there is, uh, some interesting, you know, real results here. So, um, so some teams have really taken their CICD practices, uh, all the way, um, I think, uh, in a really interesting direction. So this is an actual button, um, that if you push it, it will attempt to take whatever is checked into Maine and put it in production. Um, and this team had developed such confidence in their CICD pipeline, um, that they knew that if you punch this bus button, that their test automation, their deployment automation, uh, was, uh, something had so much confidence in that, that pushing this button at at any time, uh, was gonna be something that was safe to do. Um, as it turns out, this is also one of the teams that was an early adopter of concourse. And so, um, because they had implemented a lot of this, uh, advanced deployment automation, um, and test automation, um, in concourse, it made it that much easier for teams to adopt, you know, those same practices at the same time that they were migrating to Concourse. Um, and so we really have started to see this really pay off for us <affirmative>.
Yeah, I remember that button. I, I pushed it once they let me push it one time. <laugh>. So, so what are we looking for? Well, one thing is, um, we wanna have a conversation, especially if you're trying to introduce the same kind of commonality across your enterprise, where, uh, you have a lot of independent and self-empowered teams.
Yeah. We're, we're really curious if you're working on this, uh, what has worked for you, um, and what have you tried, maybe that hasn't worked. Um, we would love to swap stories with you and, and collaborate on this because, uh, this is, this is an important concept and a, and a problem that we think is, uh, is pretty common across our industry. Um, and I think we can solve this better, uh, by working together on it. But in the meantime, uh, you know, let's summarize some of the things that we've learned while we're trying to do this. Um, I, I think we'll start with the observation that Mike made at the beginning, uh, where, um, if you do empower teams to choose the right tool for the job, um, but you have a large enterprise that can result in that. You've chosen every tool for every job, uh, at the same time.
Yeah. And, and one thing that we saw, uh, is that the more freedom you give, the more variety that you have, sometimes that causes friction. Uh, it also prevents maybe some of your individual contributors to be able to move from team to team and take their knowledge with them. Uh, and it also can cause some issues with calibrating across teams. Another thing is, uh, again, hearkening back to the beginning, just because we can see some people can see the global benefits of standardization. It's not always that clear locally to everybody. Not everybody feels as if these benefits are gonna be, uh, uh, really be beneficial to them.
Yeah. And, uh, you know, fortunately, one of the things that we have found is that, uh, we found that taking a structured approach can work for these types of changes. So, uh, whether that is, uh, having a structured way for practitioners to participate in technical decisions like our, um, architecture guild, that, that collaboratively generates the architecture decision records. Um, you know, making sure that we take a structured approach to getting buy-in at all levels of the organization, from senior leadership to middle management, to the frontline practitioners, um, and then actually taking a structured approach to, to provide training and support, uh, to make sure that the change is successful, um, as you do it. Um, and so, so that's what we've learned. Um, hopefully you can take some of this back to your own companies, um, and maybe, uh, that will help your own transformation efforts, uh, pick up steam. Thanks. Thanks
Everyone.