Las Vegas 2019
Monorepos, Mainframes and Modus Operandi
Building a legacy application which had 9 development teams that needed to build at the same time for a successful deploy. Our challenge involved us moving to a 'super build' - which we tried to combine all developments into a 'monorepo'; including our challenges with the legacy mainframe.
PK
Philip Knezevich
Senior Manager of Infrastructure and Application Architecture, American Airlines
Transcript
So the other day, uh, I was actually looking at how many lines of code I had written in an open source project that I was working on, uh, in react, native, uh, kind of an obsession of mine coming back from college days where I would do word counts on thesis. I was very fussy to see, uh, how many lines I'd gotten. So the number came out to this, uh, 6,000 for those of you that, uh, can't see this in the back, um, 6,000 lines of code. Um, but fundamentally with all, uh, accumulated packages and components and modules downloaded, it actually came at close to about 19,000 lines of code. So my reaction was a little subtle amongst this. I just decided that was pretty good. And I decided to get my family involved on my celebration, but, uh, no seriously. Um, you know, as part of the open source community, you do inherit everyone's code, it becomes your responsibility.
Uh, you do four can check stuff back in, so you want to make sure you do things right. Uh, so tongue in cheek at 6,000, but I like to think of it more as a 19 on the other end of the spectrum. The largest source repository in the world, uh, is built in a monitor repo and it's by Google and it contains 2 billion lines of code 2 billion. It seems like the stuff of myth and legend. They have close to three and a half million commits. It's 86 terabytes in file size. It is monstrous 15,000 developers have worked on this over the last 10 years. We get nervous when there's 15 developers and one of them leaves and you know, what might happen with the code, let alone to think that 15,000 times that amount is what Google actually manage. Um, it's um, it's one of those things that you look at companies like Uber, Airbnb, Bluebell, and Twitter, they all use a monitor repo as well.
And so the idea was brought to me internally by an architect in my group that suggested, Hey, we could do this at American airlines. We could build our own monitor repo, of course, you know, from reading about this, you know, the technologist in me gets giddy and excited. And I think to myself, yeah, we could probably do this, but it's just another itch in my DevOps scratch. And I'm sure a lot of you have them here. We wanted to do it to solve things like code reusability. We wanted to do package management better. We wanted to do cross project changes better. We wanted to simplify our dependency management. So you're probably wondering why I picked three subjects as one, and I'm going to try and represent it here at the core of everything that we do is our modus operandi, simple Latin vernacular for mode of operations.
You all have a modus operandi, actual company. And over time you've built procedures and patterns and practices. And some of them may have lived for a long time. We're in the mainframe world. And some of our practices have lived back to the seventies, which is crazy. Um, but we are trying to modernize on top of that. We have all these other initiatives. We have the mono repo is one, and there are several others that we're looking at. And then at the exosphere at, at the top of all of now comes DevOps. And suddenly, as you guys know, there's the technology side of dev ops. And then there's the cultural side of dev ops. It's almost like a chemical reaction when DevOps comes in and mixes itself with processes that are just been around for a long period of time. So I wanted to talk today and tell you guys a story about the things that we went through to try and mix these worlds together.
So just before I get into my talk a little bit about myself, uh, Philip , I work in the maintenance and engineering department of American airlines, uh, 10 years originally from Australia. So I've been told my accent switches between the two over there. They think I'm American over here. They think I'm Australian or something or the other in the, in the middle. Um, I do have a developer background and yes, I have written more than 6,000 lines of code. Um, but I'm scratching my head because I came from the Microsoft world where we had fancy gooeys and we had a nice flushing things happening. And now we're back into CLA and that for me is a, is, is a head-scratcher because, you know, get know Kubernetes, yarn, Homebrew, these are all, uh, initiatives now that have just gone back to CLI. So that makes me scratch my head.
Of course, any of the software engineers in the group of like this guy's crazy, but, um, no, it really is a, you know, a change for me to understand. Um, I feel like I'm a generation back from that. Um, so just a little bit about American airlines. Um, we are the world's largest airline, uh, largest by flights, destination, uh, employees and fleet size. Um, so we fly a staggering 6,700 flights a day, a day. In that time we have to have the absolute utmost paramount in security, safety, and reliability. We have to make sure our pilots and FAS get to each flight on time, which can be challenging when they have to connect. And it can be challenging for people to get to a point when they have to connect. And I'm sure several of you may have had to connect to get here today.
We have to conform to FAA NTSB, FAA, uh, I'm sorry, TSC, uh, mandates. Uh, and then we have to work around airport congestion, air traffic, dodging, bad weather. We have to cater for many different needs for users on our flights. And as you know, people at different, uh, on top of all of this, we have to get you all there on time. And that's a difficult thing with all these other things going on. At the same time, we fly 350 destinations across 50 countries around the planet. It's a vast network. We fly to five different continents. Unfortunately we don't fly to Antarctica yet, but if we ever do, I'll be the first one to go there. Uh, but on top of all of this, we have 130,000 employees, 27,000 flight attendants and 15,000 pilots. Uh, our fleet size is in excess of 950 and growing.
This is a deliberate $25 billion investment into new aircraft and advantages offering. We have within our airline against our competitors is new aircraft. Um, we want to provide this to you and I can tell you from a customer experience, that being in a new aircraft is like being in a new car. You don't get the new car smell. You still have to smell the people within your vicinity, but everything else is pretty much the same, is that okay? So, um, internally we did make a shift to DevOps. Um, we started off with the hero developers and engineers that came into the company, people that have been around the industry that have been to conferences like this, and just we're seeing the shift and they were coming in and they were saying, Hey, we need to do DevOps. We need to do CACD. We need to validate test cases, security.
We need to do infrastructure as code, you know, all the wonderful facets that come with DevOps. And of course the people in the company that didn't know about DevOps would just smile and nod and agree and say, that's great. Go ahead and do it without really enabling them. And that had to change. So internally our CIO Maya Lieberman did make a change to bring a DevOps first culture, uh, to the airline. So we have an enterprise level tool chain now, and we have a change in our project structures to make sure that dev ops is more heavily baked and coupled with agile. Um, the industry industry is changing fast and in my area alone, we've picked out five areas that have really caused problems for us. And the first is that it is disconnected from the business and vision. What do I mean by this?
I think developers level for corporate office headphones on listening to Metallica. All I've been told now, it's it's Enya and relaxing music, but you know, keyboards, two monitors, coffee code monkey, right? They're writing code. This code gets passed through. So certain groups pass the parcel type of thing until it hits our end users in our world. In my world, it's a mechanics and line maintenance stuff and stores clerks. And, and these are people that are in airports where there's jet noise of planes taking off and planes landing and screeching and scurrying and machine machinery going on and lots of stuff happening. And the computer specs are very different. The product experience from here from the corporate office is still something that we are trying to bridge and get better with involving our business in our agile process, a lot more and trying to get better feedback from the customer experience.
The second one resonates with me a little more deeply, which is that leadership is tracking activities and not the results. And what I mean by this is a initially I was a bit obsessive and compulsive about making sure that agile rituals were followed properly. You must do your daily. Stand-ups, you know, you must have backlog grooming and you have to have acceptance criteria. You know, all the things that you do within agile. Now I've come to learn that teams know themselves better than we do thankfully. And they actually know what works better in their culture. So now I step back and I look at the results and through the results I use, the magical MTTs that you've all heard meantime to debug, meantime, to repair, meantime, to implement, and there's variations, meantime to break, fix, et cetera, et cetera. We look at how fast we're tracking and how much better we're trying to improve from a time perspective.
The third one was the other one that I mentioned that we had to change, which is our project structure, which was fundamentally broken. Funding was typically done by divisions. And you have several divisions in our organization, and I'm sure a lot of YouTube do too. And within those divisions, you get a certain amount of money from finance, you B um, and then you have to figure out what you were actually going to do to actually get done. So of course, everyone knows that, um, if they don't get their project approved for this year, it may have to wait a year or even longer. So the list would often look like this priority one priority, one priority, one priority, one priority, one priority 10, okay. Finance tools align here. But what about all these priority ones down here? It's confusing. Um, so now, you know, we're moving into the product model.
We have a recurring and ongoing backlog now, so we can just put it in there and just keep those priorities done. As we see as the most important, um, problems, four and five were interesting for me. Um, I have a lot of stories around this. I think one I'll share is a lean coffee session that I was facilitating a while back where we had 12 different people from 12 different groups. If you haven't done lean coffee before I strongly advise you to go do it, it's a great experience. But we got stuck on a topic. What that was basically business values and it values do not match. And everyone, all 12 of them actually agreed on this as the number one problem they had in their group. One great example was a guy who had actually said that there was a spelling mistake made on a simple HTML page, right?
HTML page, simple spelling mistake. Couldn't get into prod for two weeks because that change was bundled in with all these other technical debt changes and nonfunctional changes. And suddenly they had a release cycle and they had to get all these other things approved. It took them two weeks to have a spelling mistake there. I don't know what the word was in the end, but, uh, that's interesting. So, you know, as soon as, I mean, it has often been around for a while. I've, I've collected these memes, um, many times over, um, I'm sure a lot of you are familiar with them. The one on the right came out this morning. Um, but you know, this, these are the types of memes that you see, um, you know, being socialized across a lot of technical groups. Um, the one that always gets me well, there's two is the networking guys cause they always get blamed first.
Um, and then finally, when, uh, you know, when the PM's and the managers try to work with developers for estimates and then just end up using them as deadlines and the conversation typically goes, you know, how long is it going to take? And the developer, you know, kind of defensively says, well, if I had a requirements document and you know, no unplanned work and I had, you know, that gun developer helping me, you know, plus minus 30% take three months. So the asker, the PM will say, okay, three months. Good. And then that's what they use. Like it's, it's crazy. Right. And both walk away feeling a little awkward about that conversation. Um, so internally we, we were trying to ratify these problems. We're trying to get away from them. Um, so just a very simple workflow here in, in my area. Um, you know, we have our UI, our business people in our case, it's the line maintenance folks.
It's the people at the airports working in maintenance and engineering, typically just interface with a UI and then behind the scenes, they don't care what happens to us for them, but behind the scenes, you know, we typically go to a solar layer, um, which then goes to an abstract suffocation team because we don't allow direct access to the mainframe. The problem we have is that each of these, um, uh, silos have their own teams. And this is where it gets tricky because they have their own managers and their own priorities and their own deadlines. The solar team in particular gets more messy because they have to deal with, um, other groups that also have their own managers and teams, and also have to work around their priorities. And those people will push back saying, I can't do that because I'm also working with all these other app teams.
So the lines of convolution and messiness start sprawling all over the place. It becomes very difficult. And then the worst part of it all is just trying to work with the mainframe team who are also working with all these other app teams, very confusing. We had to think differently around the way we were doing this. So as you're all in, as you're all probably aware, there's this notion now of products moving away from projects to products. And so what we did was we quite simply just took out the resources from those groups and we created this notion of a product team. Now we're still in our infancy with this. We're trying to get better with this, but, um, fundamentally, uh, it's, I've been, it's been described as a get out of my way to get out, to work my way architecture. Um, it's a small business model.
Think we announced small business team. You know, we, we have the resources we, we want from each of these groups. Um, and so then now, you know, incubator design self-sustainable, we don't care about the rest of the world. My product team has it. All right. So it's kind of trying to bring that speed to delivery in that one of the challenges we have with this, and I should also add, we've also included architects, SRS, um, you know, some QA UX. They're also part of these product teams in this self-serve sustainable environment. One of the problems we had is, you know, now that we're trying to move to dev ops, we were having all these multiple pipes, proteins and are having their own multiple pipes. But we also have a monitor repo, which I'll get to in the moment. Cause I want to talk a little bit about the mainframe.
I say this funnily, they're not really the black sheep of the family. It's just, you know, you go on, you, you go to Google and you, and you DevOps, you search for DevOps and you'll get hundreds of examples in the JavaScript, java.net world mainframe, not so much because they've been around since most of you were born. Um, but you know, there is now as you saw wonderful presentation in the morning from CSG, there's a lot of work being invested into diversifying the mainframe. And we went down that route as well. Um, so we started partnering up with a company called Compuware I'm sure you've heard of them. Um, Chris O'Malley and Greg gave a great talk. I think it was in London. Um, and so with Compuware, um, the first thing we did was try and get our source control, um, into a proper repo.
So we using ISP w for that, it's a little blurry from my side. I don't know if it is for you, but we are using, um, ISP w um, for, for the, uh, promotion and build of code. Um, then we use, uh, Topez total test to execute and validate our test cases. Um, and then there's quite a bit of analysis gate check type stuff before we finally, uh, promote to our integration environment. And then IFPW helps with that. Um, I'm not going to go into the same detail at CSG did this morning, but, um, fundamentally, um, we started making sure that we had the right checks and balances in a lot of the, uh, mainframe code that we were writing in Qubole, um, where these got challenging for us is the mainframe is a tightly coupled dependency to other systems. If a field changes in the mainframe, all the other subsequent downstream systems also have to react to that. So we wanted a way to try and put that into our Jenkins pipeline and bundle it together. And for that we're actually using, um, XebiaLabs um, and they have a tool called release XL release, which actually orchestrates if we have any JavaScript code or, or, um, any dotnet code or any Java level code alongside the mainframe, it'll actually build it all as one.
So I'm going to change gears for a second. Uh, let's talk about the toothpaste. Um, anyone who has kids knows kids love to squeeze toothpaste at the very front and they'll just squeeze out it and squeeze it and squeeze it. And then they're like, oh, it's done. And then they'll throw it in the bin. Sure. It drives you all nuts. Personally drives me nuts. I've got to, and they do that. And I have to pull it out of the bin and say, look at all of this. But anyway, um, so when we were in the situation of writing test cases, we, we had a very similar feeling to this. So the main frame is not an object oriented framework. It's, you know, COBOL is not that COBOL is, uh, prescriptive mainframes typically rely on three letter transaction codes, which in turn interact with what's called an MFS or a message file system.
I think of it as a glorified parameterize. Uh, um, and then it interfaces with PSPS, which is a logical IMS data structure, which gets represented as a program. Um, so initially at first we were writing these tests and we were squeezing up the front and the test came out. Okay. And then we came into the situation for the mainframe knowledge is in the crowd called the conversational transaction. And the conversational transaction is pretty much what it says it is. It's you having a conversation with the mainframe? Well, you'll send inputs to it, they'll send a result and then you can make a decision around that results, send it back, and then it can happen a few times and you have this decision tree. So we started to realize a lot of the test cases that we were writing, um, were not inclusive. A lot of these use case scenarios and, and decision forks that were being made.
So we were stuck in the situation where the tool, which was meant to be a slave for us was actually more of the other way around. So let's try and automate it, you know? Um, so yeah, sure. This will, uh, this will make it happen faster, right? You just wrap a bit of court around and it'll it'll happen fast, but the same problem, right? Once you start investing in, um, in automation, um, you know, you can do things quicker, but you may miss a lot of these things. Um, so we had to think about this. Uh, we partner up with, uh, a company called DXC. You may know them and actually normal. Who's sitting normally want to stand up for a second. He's our guy that's been putting a lot of this together. Um, we, we really had to think methodically about what we were going to put into our load testing tool.
And, and after this, don't talk to me, talk to him, I'm going to defer it to him. He'll, he'll tell you some stories around how we tried to set that up. But, um, the fundamental, the story is, um, you know, don't, don't automate too much, um, with fast fancy tools and, and this may be applicable to you guys. Um, maybe this is not a true statement for you guys, but for us, we found that we could have gone to Amazon and purchased $120, um, toothpaste, extractor. And I apologize if anyone has actually bought this. Um, but you know, in a situation like this, um, you know, toothpaste is limited to certain forms of Jube. It's harder to clean and maintain it. It has unnecessary dependencies, it's, it's embedded in the wall, right? So, um, and we felt that way with a lot of the thinking behind automation, um, in testing, I, I'm more thinking of this, you know, back in the old days of Merkel mercury, low runner, where you had to isolate your own environment, you had to get test data in there, and it was an environment you couldn't use QA stage or any of the other tiers, because the purpose was to actually bring it down through capacity testing, but it was kind of a kin to that.
Let's think about what we're actually going to try and do without testing. So in the end, the $8 99 tool from Amazon, the little glider that comes to the back is kind of more a metaphorical for what we were trying to do. Let's take a step back. Let's look at the process. Let's promote the notion of inner sourcing, open trust and collaboration between developers, for communities. We try actively to get our mainframe is to speak to our technical guys. We're trying to really promote this idea of making sure that chess coverage becomes an integral part of what we do, you know, trusting in the fail fast process of DevOps, because what happens is every developers. Number one thing is let's get the damn code into production. I don't care about test cases, but failing fast means that you actually learn a lot quicker and you don't learn from prod mistakes.
You fail in the environments that get to Prague. So we had to bring that mindset back into a lot of the thinking of what we were doing, um, fundamentally, um, through DXE, um, and normal probably correct me on this later, but we actually, um, we estimated we do save about 30 hours per build via automation. Um, so I don't know norm if that's right by you, but that was what I was told. Um, so the overall stack, um, that we have, so we have to, we, uh, we use on the mainframe side, we use IFPW, we use Topaz and then in the apps ops intelligence, we use Compuware and Z adviser. Um, on the more traditional side we use, uh, GitHub enterprise Coverity, um, and then it's it forks between ado and Jenkins. Um, and then as I said, uh, XebiaLabs XL releases where we bundle it all together. Um, we have Moogsoft, which we use to aggregate a lot of our logging into one easy to view place, um, and then Dynatrace for our runtime or time, uh, troubleshooting and debugging.
Okay. So another change of gears, um, I'm going to talk about plumbing. I don't know how many of you people here are plumbers, or if you've actually ever seen a wireframe of plumbing in your house, but it kind of looks like this water comes in one pipe and then typically hits your hot water, and then you have two pipes and those will probably disperse around your house. You know, kitchen, laundry, wherever you, wherever you need hot water. Um, but then typically it always goes out through one pipe. And so the fundamentals of this is that, um, when we look at the way that software has evolved, um, you know, from the evolution of TCP to sub services, then we went to wrestle API APIs. And now we are looking at microservices and there's all these changes that are happening now in the way we, we write code.
But the fundamentals of it all is that you hope you're doing the full principles. You're, you're building, you're compiling your testing and you're releasing code. So the monitor repo takes this facilitation in mind, thinking you're here and you need to get to here just like the plumbing, but everything that happens in between the mono repo believes it can handle it all in one goal. So I couldn't get this, uh, in my team as a, as a jingle that we could go along. But, you know, I substituted the word ring for a lot of the rings for the monitor to describe the, a reboot it's one bill to rule them all, one, built to find them one bill to compile them all. And with DevOps bind them, you know, I couldn't get, I couldn't get it caught on for some reason. I, you know, I'm not sure why, but I'm so simply the monitor refer works very similar to it at a traditional SDLC.
You sync your workspace to your repo the same way, except now you've got everything. Uh, you write the code the way you usually do. Um, and then you just check it in and you review it. I'll go into that process a bit more. Cause it's not that simple and neither is the commit. So just coming back to my original point, could we actually do this? It was asked the question, um, to me and I said, well, why don't we try a subset experiment? Um, we said, we we'd pick 15 apps. And in there we would. Um, and we, we decided the UI ones first because they were Java script. Um, they had a mature DevOps stack. Um, and then we actually now have some Java and.net in there as well. Um, so we decided to do that and we decided to compile it with 25 shared packages at, uh, intrinsic to American airlines.
Things like connecting to our flight services and, you know, cargo and maintenance services, those types of packages. Um, and then we actually have 90 plus third-party dependent, third-party dependencies that are being used. Um, this is stuff in the NPM world, commander react, express sync, body, passer load dash, you know, for the, for the JavaScript is in here. You'll know what I'm talking about. Um, as well as angular seven. Um, and then we started off with 70 developers and, and this is quite an interesting experience because some of those developers had been working in one product for 20 years. Didn't know about this whole other world around them of it, right. Just I do my job and I do it here. Now, I suddenly saying to them, you need to bring this in and you need to own everything that we actually do. Cause our goal was one CIC pipeline.
Um, we do have targeted build and release, which I'll get into in a moment. Um, but I've tried to simplify the view of the monitor repo here for you. Um, we have apps that just sit in a sub folder or the way that they've always sat. Um, the difference is you'll see here we have node modules, build scripts, just libs tools. You know, there might be a bin folder. There might be an etc folder. They typically exist inside the app structure. And there's usually 15 of those. Now we've moved those into a shared space. And now we expect our app teams to now reference in there, um, and, and use them as necessary. I like to think of it more as include files from the old days, but I've been told it's a lot more complicated than that. Um, but so essentially what happens is if there is a module in the shared area that gets changed by say app two, it may affect at 4, 7, 9 and 14.
So now app two needs to make sure that whatever he or she has done will not break anything in those other apps. So they check their code in and we now need to, um, check it against everything else. If we had a more TDD, mature environment, this might be easy, hell, I'll just run the test cases in. If our test coverage is good, it's there. But a lot of these apps are legacy. They don't even know what TDD is. Um, so we've really had to try and do a lot of things to fix them on our repo, to be more fluent to it, to deal with these situations. Um, so we started looking at tagging. Um, so every single application is actually attributed to certain files and then everything runs off this master trunk, right? It's in get hub, everything has a master trunk. And then for targeted builds, we just do cherry picking.
So we say application server needs these modules and these dependencies, and that's what creates the build for that app. We define our app through quite simply, it's just a Jason file. Um, so that cherry picking, um, go through and we've actually gotten smarter, smarter with this, with our tagging. We actually know which files affect, uh, which applications directly. So as soon as you edit it automatically, we will build out those apps that are associated to that. It is scary. Um, it is scary because you don't know a lot about other people's apps. Um, so from that perspective, the number one fear everyone has is, um, let's, let's try and avoid using that as much as possible, but there are some instances, there were some brave ones out there that said, Hey, I want to upgrade from angular six to angular seven. And they had to do it for all the apps.
And that's where the value of something like this comes in. So just back to my old programming days, um, this was a stress for me, you know, 12 o'clock at night, you know, 6:00 AM deadline, and someone's making a massive change. I'm like, don't break that build, you know, you kind of get worked up about it now suddenly it's not just one app you're, you're responsible for everything in the entire organization. Holy cow, how do you manage something like that? That's where our check-in process has become very thorough. We have queued up several check-ins, um, and we analyze them very thoroughly before we commit them. Um, so if some things take days and we've left them there and some things are so old, we abandoned them because they've been superseded by new changes or that developer has a new check-in. Um, so we don't want to be stuck in a situation where we just keep ignoring stuff cause it's taking too long. Um, so we've been very intensive about making sure our code reviews have been more thorough on this side.
So resulting to this, and I'm sorry, I've realized I'm a little over time here. Um, consolidated all our development into one source of truth. That was the big win for us. It's all in one place now. Um, we do code sharing from legacy apps, uh, reusability, simplifying dependency management. Uh, there's been a massive increase in development teams, whether you like it or not. It's scary if an app has no developers, they're, they're out on vacation and no test cases and you're making a change into their app, right? So they've actually learned to speak to each other a lot more than they used to. Um, we've got new developers coming in. I said we started with 70, I think we're at about 95 now. Um, and then just realizing the benefits of a write once, use many, you know, don't repeat yourself dry versus wet. Don't repeat yourself versus write everything twice.
You know, we're on the dry side. Um, so result lies 330 times faster to deploy these dependencies. This is a big win for us. Anyone here that has a lot of technical debt, a lot of ESL type of upgrading this. This is a great solution for you guys. Um, you know, 250% increase in developer collaboration. We see it a lot. They, they definitely are, um, interacting with each other a lot more. Um, 4,200 workers were gained by eliminating the duplicate efforts of a lot of the package management. Um, this is a big one for us. I think there's what 16, 1700 hours work hours in a, in a year. So that's close to two and a half years. So finally, um, one of the challenges we had, as I said, TDD is a problem. We need more help with TTT, struggling with feature toggling. Um, I wish I had more time here to speak about feature toggling.
It's a fascinating area. Go see LaunchDarkly if you can. They're great. Uh, we have our own solution internally. Um, we still rely on a lot of the companies that helped us with the monitor repo. One of them is a NOAA, which is a Google X. Google is at, um, uh, specialized in the monitor repo. They came in to help us, um, cutting the cord with them. I don't know if it's in sink or swim. I don't think we're drowning, but I don't think we're swimming. We're kind of in that middle place. Um, and then the biggest problem is most of our projects are still in the traditional Polly repo 15 apps in my area alone. I own 330 and around American airlines in general. Um, thousands to, could we ever expand this to the larger org and become a, you know, a true monitor, repo like a Google or an Airbnb?
I don't know. I'd like to, to pipe dream. It would just require a lot of cultural change to make that happen. So my final slide, um, three different topics, one destination moving to one repo, one pipeline, which also includes the mainframe. That's our goal. Um, if you're thinking about trying to do some of these things in your company, you try and understand the right approach. Your teams have to be transparent. You can't have cowboy developers sitting in the corner going, I can do this better than anyone else and trying to prove that they're better and doing it on their own. You really
Need to have that, um, extensive collaboration, which begins with trust. My name's . You can get me on a big apple PK. Um, I, my phone likes to act like a bird, please tweet me. Um, and I hope you enjoyed my chat. Thank you.