Las Vegas 2018

The Delta Air Lines Dojo Journey

Delta launched a Dojo in June to bring together six transformation efforts to work directly with delivery teams to change the way that they deliver software. We have coaching representation from Agile, API, Cloud, DevOps, Quality, and Security in a dedicated facility for a duration of time. In this forum, we will talk about our engagement model before the Dojo, address challenges leading to the Dojo, the tweaks we have made, and our expansion plans.


Tasked with leading DevOps Transformation at Delta Air Lines in 2016, he launched a Dojo effort in June of 2018 with peers from Agile, API, Cloud, Quality, and Security teams to change the way that software is delivered at Delta. Before this role, Keanen has acted in various roles in development and infrastructure teams which is helping to bridge the two organizations within Delta Air Lines.

KW

Keanen Wold

Manager, DevOps Transformation, Delta Air Lines

Transcript

00:00:03

My name's Kenan Wald, and I'm here today to talk about Delta's Dojo journey. And, uh, I realize it's the last day of the conference. Sponges are kind of full, um, excited to probably get home to your families and, uh, you know, better yet get into the office and implement or get working on some of the great things that you've, that you've learned here in this wonderful community. So I thank you very much for your time today, and, uh, thanks so much for, for showing up. Um, let's talk about Delta Airlines. So we generate $40 billion in annual revenue. We turn 90 years old next year. Been around for a while. We have over 80,000 employees use technology pretty much since it existed various ways. Um, and technology's recognized for our company as a top priority and everything that we do, we try to be thoughtful, innovative, and reliable all throughout. And, you know, you can see how those themes really resonate with what we do collectively in, in our technology space.

00:01:09

This is an interesting slide that came out our data point. This year, Forbes announced the top our most relevant companies in America. And when you look at this list, Delta's number 11 between Netflix and Spotify. What I find that's very fascinating about this is, you know, general day-to-day conversation. What Tesla's doing to the automotive industry. Uh, you know, hey, I'll send you my Spotify playlist to get some great tunes. Did you binge this weekend on Netflix? Like just water cooler talk and, you know, the, the brand presence that these companies have. And to be there, I think, you know, we're very humbled and honored. But what really sticks out when you look at this list is how young many of these companies are. There's only two companies of significant age here. Disney's the oldest. Disney Brothers, started in 1923, and Delta started flying customers in 1929.

00:02:10

We we're crop dusting before that. So we check Wikipedia, it'll show a little different. But, um, you know, like I said, we're hitting our 90 year anniversary next year. The next closest is Nike started in 1964. Um, so comparatively, you know, the age of these companies and, and for us to be on this list and be relevant, I believe because of, you know, what we do in technology at Delta, along with our customer experience, and a lot of things we pride ourselves in, uh, is just very fascinating. So we're very proud of that. We know to stay on this list, we're gonna have to, you know, continue to, to push on this stuff. And we've had conversations this week about how companies have, you know, fallen off, um, because they haven't had good technology strategy. So we know that we, we have to keep pushing forward about myself.

00:03:03

So, um, I'm coming up on my 18 year anniversary this time around. I, I started at NWA, um, as an intern. No, I wasn't hanging around at Dr. Dre and Ice Cube. This is Northwest Airlines, <laugh> Northwest Airlines. For those of you that remember us, we're actually coming up on our 10 year anniversary, uh, uh, since the merger. And I've played various roles. I started out as a COBOL developer, had to hide that on my resume because you're getting like 18 calls a day in the late nineties. Um, moved into distributed development, moved into a, uh, infrastructure engineering role as a web engineer, did some WebSphere work, moved back into the portfolios on the app dev side as an HR architect. Went back into infrastructure as a leader and found myself in, in this role that I have today. The, uh, much like, you know, many other people at this conference and, and many of you, you know, um, we, we find ourselves in a position of having empathy because of our experience. And, you know, no different than so many other people that have spoken at this conference. I feel that that, you know, purposely bouncing between, uh, the sides of the wall, if you will, within my organization, uh, has helped me position to be, uh, you know, in this role.

00:04:26

So my team, um, I manage the DevOps Center of Excellence and Developer Practice. I'm also the Dojo Program sponsor. Um, my team started to form in late 16. I started to, I, I still had my infrastructure day job as a, as a leader there. And I started pulling in resources and we were trying to figure out what we're gonna do to get going. Um, one of the first people I pulled in is Caleb. He's in the back of the room there, Caleb Wolf from Delta. Great, great guy. He's a, he's a wonderful guy to talk with. And, you know, I, I started pulling in like-minded individuals who were gonna help change the way that we deliver software at Delta. We started by, um, selecting some tools. We needed a baseline foundation to what we were gonna do to launch the behavior that we were really intending.

00:05:20

So my team has three responsibilities, those developer tools, anything the developers touch IDs, um, you know, we had legacy source control. We launched GitLab. And by the way, thanks to GitLab for, uh, for their encouragement to actually do this. I wouldn't be up here today if it wasn't for them pushing me. So I appreciate that. Um, we implemented Jenkins, Sona type, nexus, sonar, cube and Slack. It's basic foundational pipeline functionality that would get people enabled to do the, you know, work that in the way that we, uh, desired. The second thing we're responsible for, for is a developer practice. So I've got this, uh, advisor, architect, uh, Encore on my team, and he is doing a great job. Uh, our, our CTO KK has enabled us to do some, uh, bits and bites sessions. We buy lunch on Wednesday, we buy pizza, you know, we're bringing people in. It's, it's building a community of practice where they can show off the things that they're doing. We can show them, you know, where we're trying to take things. And, uh, you know, he does a great job there. And then the last one, the third one is the, the DevOps cultural adoption, which is carried through in the Dojo and some of the other engagements that we do. And of course, is the hardest one of 'em all.

00:06:37

So, technology footprint, you name it, we have it at Delta. Uh, you know, operating systems databases. Uh, we're heavy Java shop. We, we, we have a lot. Um, here's a picture, a couple pictures from our museum of our, uh, first mainframe implementation of the sixties. Apparently you had to wear short sleeves and a black tie, uh, in order to be around that equipment. Um, <laugh>. So yeah, the, whenever you guys use delta.com or Fly Delta and, um, you're, you're booking a ticket, you're changing your PNR. There are service layers that go down and talk to our, our ZTPF system transactional processing facility. It's based on c Assembler and SABRE talk, you know, and we have many layers, uh, beyond that. And again, you know, this is what you use today, talks to the, the mainframes. But, you know, we have to deal with that. When you talk about the, the cross-functional nature of the work, it, it's, it's vast for us because there is such, um, you know, so many layers of, of technology at Delta.

00:07:44

Okay. All right. So why did we need a dojo at Delta? So, Rahul Samant, our CIO joined us in February of 2016. And, you know, he kicked off some transformational efforts. He wanted an A-P-I-C-O-E. He, he wanted an agile COE my DevOps team. Uh, we already had quality and security, and, uh, we're introducing a new, uh, private cloud platform. So we have all these teams and they're engaging with, with delivery teams. You know, if, if somebody was using a, a pipeline, maybe even in survival mode before my team had it available, they stood up their own Jenkins instance and, you know, what have you, uh, they were off doing things. Maybe they felt good, like, okay, I've got the pipeline thing down. I'm doing the DevOps, and, uh, you know, hey, maybe I need some help with my velocity, so I'm gonna engage the Agile team, or I need some API design help, so I'll work with the API team. Um, it was disjointed, so it was like, it's kind of a la carte for these teams, and it didn't create consistent delivery or behavior, uh, for the teams.

00:08:52

So we, uh, our, our journey begins last year in London. We had, uh, Jasmine, who's one of my many rockstar engineers, and Cassandra on the right, she's my peer that runs the Agile COE. Uh, we got to meet the, the trio here. Uh, love these guys. They've, they've all been very helpful and, uh, resourceful and, uh, you know, started to build this, this network of people who are very willing to help. And I hope that you guys are really embracing that here in this conference. If this is your first time, um, there's a lot of people here who are willing to help and, and exchange stories, um, make sure to, to leverage that, reach out and, uh, you know, please, I'm, I'm happy to help out as well. So we, we, we had that, that conference, and it was very motivating and, and enlightening to, you know, other people are going through the same struggles.

00:09:49

Um, you know, we're not much different from other companies. And, and I think that you guys find here as like a good, uh, therapy session. It's, it's, it's very interesting how likely we all of us are. So on a timeline, you know, like I said, agile, COE, they were engaging, they were coaching teams. Uh, they were the first one that was stood up. I was still building my team in late, in late 2017. Really started hitting the ground in, in, um, early, early 18. And then, or I'm sorry, in early 17. And then, you know, we started to go talk with other companies. We, we met with Ford, we met with Home Depot. We learned about how they, uh, approach software delivery. We talked with them about their private cloud platform. And then, like many others, uh, you know, we went to Target and they were gracious hosts.

00:10:37

My neighbors up in the Twin Cities, um, got to talk with those folks. Beautiful facility. Uh, they taught us about, you know, how they're succeeding in their dojo delivery, what their dojo is to them. And that really got our minds going. Uh, you know, uh, around the time that we went to the, the DevOps Enterprise Summit conference. So in the fall, we started to strategize around how we were going to, to do this. I pulled in a partner called point B. Um, they've been extremely helpful based outta Seattle. And we started to drive our strategy for a launch in June of this year. Uh, and you can see that, you know, we continue to look for input. We meet, recently met with Honeywell. They're doing some really cool things, um, in, in the Atlanta area. Uh, it was very exciting to see some of the things they're doing.

00:11:28

So we get our strategy kind of put together. We needed three things. We needed a physical space. We had this large effort going on, uh, in ano another area of the, the IT organization. And they had this huge room about a quarter mile off away from our headquarters, more like in an operational building. And, uh, space isn't easy to find. I was really jealous of the target trip because, man, they have a lot of space. Um, so I, I see this room, it's surrounded by kind of five smaller rooms. Um, and I'm like, Hey, we'll, we'll take it. And we take that room and we, we line up tables and we, we set up tables. There's six of 'em to support six two pizza teams. We made it our own. We put some color in there. Uh, you know, presentation capabilities, uh, you know, better collaboration capabilities between the teams.

00:12:22

So we've got our space, that's number one. Here's a classroom that, that we use, and we use this to, you know, promote behaviors, the benefits received when, you know, following through with test driven development, continuous improvement, um, you know, quality mindset and, and secure coding practices, et cetera. We'll do workshops. We do formal training, you name it. But it's nice having that in our facility. So we've had the physical space now too. Uh, number two is a set time. And what we decided on, which is a little bit different than some of the other dojos is some of the other ones are doing more accelerated muscle memory building, which I think is wonderful. What we decided on is, uh, four two week sprints. So we have teams in for eight weeks and, and we help them. Um, you know, we, we get them in there. We teach them, uh, about, you know, the cadences and, and everything. But it's, it's more real time. And when they're there, they're not. They're for a, a training. They're giving up some of their time, and it's immersive experience. We are working along with them while they're, uh, delivering their product.

00:13:35

Number three is the coaching staff. So we've got, again, all these teams, and you can see now the arrows are kind of turned around and we're working together, um, you know, with the delivery teams, and we've got some extra support. Rahul often talks about fingers to a fist. So we had all these disjointed efforts that I talked about earlier. How are we gonna bring that together? And that sounds kind of violent, not intended to be, it's a gentle fist. But, uh, you know, the, the coaches, again, they inject themselves into conversations and ceremonies. They may offer some help or guidance, like, you know, in line, they might pull somebody aside afterwards, would you consider this approach? You know, uh, here's another, uh, way to do things. And, you know, it's, we are there for them when they need us. They come and physically grab us.

00:14:25

We had an incidence in the first wave where it was a Tuesday, and a project manager scheduled a meeting for a Thursday. And I, I was kind of baffled. I'm like, we're right here. Just grab us. So, you know, again, that muscle memory or that behavior and, and changing that and, and teaching people that you're accessible and you're there and you know you're there for them, uh, it's, it's very enlightening to them. And it's fun to see as they start to grasp that. So, again, the tools are easy. We're promoting the behavior, um, just, you know, continuous improvement automation, debt reduction within sprints, empowering the teams. If they all come in with varying levels of maturity, they might be waterfall. And we teach 'em how to get into a sprint cadence and conduct those ceremonies. They may or may not have ever done API design. They may be on our pipeline. They may not. They might be extremely manual in their processes. We'll help them with the automation. So again, we're just pulling on all these levers we don't have any qualification for. Well, you have to be up to this bar in order to move forward. It's, you know, wherever you're at, we're gonna help you get better.

00:15:34

Um, another thing that we had to do is, with that eight weeks of time, is we have what we call a concierge service for some of the infrastructure, uh, for some of the legacy processes, we have ServiceNow requests, and then things fall into a black hole. Um, being that we only have eight weeks for some of those longer requests that might take two weeks to, to get turned around, we don't have two weeks. So what we do in these intake sessions, when we get, we're getting to know the teams, we go and grease skids with infrastructure teams, and I talk with my peers and, you know, we, we get resources allocated. So during that eight weeks, we still follow the process to, to do the request, but then I have like a bat phone. And in some cases, we just have the people come live with us in the dojo for infrastructure support.

00:16:23

So here's, uh, coaching staff. We're missing one guy, Chris, from my Agile team. He was, he was out sick when we were taking these pictures. But these guys are the heart and soul of the dojo. Uh, they do an awesome job. And, uh, you know, I'm, I'm blessed to be surrounded by such talented and passionate people. Um, I, I love working with these guys. And, you know, they're the, they're the real ones that deserve the praise from the, the teams and our leadership. So, um, we, we got those three things, physical space, set time, and our coaches aligned. Now we had to go and put our marketing hats on and go sell. So we went and talked with, we have three, uh, delivery VPs. They all happen to be named Matt. So, so yeah, Matt, Matt times three. We went and spoke with the Matts and, uh, you know, kind of sold them on the concept.

00:17:16

And, and, you know, some of them were kind of more aligned with where we want to take things. Uh, some of their organizations were not. Um, but, you know, we started to build our own backlog. We conducted intake sessions, and we still do, that builds our backlog. So every week we invite teams that are interested to come in, and we have conversations and we find out where they're at in their journey. And then that, you know, helps us kind of prepare for what we're gonna be helping them with. We ask that they're long lived, dedicated teams, two pizza teams, is what we're looking for, um, and that they're interested genuinely in improving the way that they work.

00:18:01

So we get that list and, and lift off. Now, this was an experiment. I mean, you know, we didn't, uh, while we started looking at this in the fall and then carrying up till the summer, uh, you know, we had plenty of things that we're working on and, and getting some formality around the program. But, you know, as June 4th was coming, people were getting nervous. Uh, it's may, uh, middle May, and they're like, man, can we push the date on? I'm getting kind of worried about this. I don't think we're ready. It's like, no, dammit, we're gonna move forward. And, uh, you know, we figured it out as we went. The facility wasn't complete. You guys saw the contrast in those, in those pictures. And, you know, for eight weeks we worked with three teams. So we started out small. Everything we read says, you know, start small. Don't go crazy. Um, and it was a lot of fun. Eight weeks later, we graduated those three teams.

00:18:58

Um, that was on a Thursday at the end of July, Monday morning in August, we bring in six more teams, and we graduated them at the end of September. Uh, we're now in our third wave of our dojo, and we brought in six teams to Atlanta. Uh, we on the 8th of October. And in addition, we actually brought in a Minneapolis team. So we're very excited about that. We're, we're starting our dojo in Minneapolis. All right, so victories. Um, we had respectable demos after only eight weeks. We, uh, we've had production launches from the wave one and two teams, um, since, since they've gotten out. Um, that's awesome. High fives in the hallway. You know, they're so, so excited. Hey, man, we pulled this off. Uh, customers are happy. Um, through that concierge service, we're starting to bridge and, and find low hanging fruit with the infrastructure organization.

00:19:56

You know, hey, uh, what, what opportunities do we have to improve this process? What opportunities do we have to, you know, automate, uh, some of the things we're doing there? And, you know, seeing the mindset of the teams questioning how things are done, or why they're, they are that way is so rewarding. They, they're constantly challenging things around them that maybe in the past they would've been complacent with. We are, you know, I kind of, I, I tell 'em when we launch the thing every time, like, we are the anti, this is how we've done it for 10 years. Like, let's, let's get beyond that. Let's, let's, let's make everything better. Um, so we have a, a question that we ask a survey. It's like an MPS thing. We say, you know, would you recommend the Dojo program to your peers at Delta? Wave One said, 87% said yes. Wave two, 89% of the graduating team said yes. So, um, you know, that, that was very exciting for us. I, I really wasn't sure what to expect. Thank you, <laugh>.

00:20:57

Thank you. Struggles <laugh>. You guys ever done a jet pull? It's pretty fun. Once you get it going, it, it's going <laugh>. So, uh, there was a couple times I wasn't sure that we were gonna move forward. Um, if there's anything that you guys take away from this particular presentation, if you're preparing to take this journey on your own, just recognize you're gonna get punched in the kidneys, uh, just when you think you're off and running. So I, I was asked at one point to kind of put things on pause and, and kind of, you know, we'll, we'll assemble the band again in a couple months. And we ended up pulling through. We, we didn't, we didn't pause it, but I was just disgusted about it. And I had my head in my hands, and, and one of my buddies walks by, he goes, it can't be that bad.

00:21:44

And I, I'm kind of an emotional guy. I mean, I, I think I was probably kind of welling up a little bit and trying to hold it together. And, um, you know, we ended up, I have a call with, uh, my friend Nick Whitaker at Virgin Atlantic every once in a while. And just so happened that that day we had a call. So I get on the phone with him, I'm like, man, I can't believe this. I thought we were golden. And you know, he, he's like, Hey, my mentor at another Virgin division told me this is gonna happen. It'll happen to you, it'll happen to me. It happens to everybody when you're trying to drive this kind of cultural change. Just when you think you've got it, you find out you don't. And so that really helped me because I realized, well, okay, I was being really hard on myself.

00:22:27

Like, what? How could I avoided this? You know, how could I had done this better? Or is there something wrong with Delta? You know? And, and it's like, no, this is just something that happens. So, again, if you learn anything from this presentation, if you're gonna take this journey, be ready for the fight and keep fighting, it's worth it. You know, it's, um, it's worth it. Walking into the dojo and seeing the buzz and the energy and the excitement from these teams puts the biggest smile on my face. There's nothing more rewarding than seeing something like that. So, you know, like I said, keep fighting. It's worth it. Lessons learned, three main ones. Um, all teams are different. Be prepared to be prepared to adapt to their needs. Uh, we've heard some presentations this week. You know, if you're overly prescriptive, you're gonna kind of box yourself in.

00:23:19

You can't expect the people to form you. We're, we're coaching, right? I think the best coaches play to the strengths of their players. You don't make your players kind of play to your playbook. Um, so, so be ready to adapt. Number two, um, be careful of hard deadlines. We learned this in wave two. We had some teams that came in. Again, this is an immersion experience. We need time to coach. And if you're stiff arming us the whole time, time, because you know, you, you can't take advantage of the coaching time. It's not working for anybody. And it wasn't the fault of these teams. They were told that they had a hard deadline and they were trying to deliver on that. But it's something now that we look out for when we do these intake sessions that perhaps they're not a good fit at this time.

00:24:04

The idea of slowing down to speed up, I mean, that, that exists in this scenario. Number three. Um, we have a lot of work to do in the, in the housekeeping of our eight weeks with our teams. So, you know, we continue to iterate through that. Between wave two and three, we gave ourselves a week. So that wave one to wave two is like, bye-Bye on Thursday. And then welcome the new teams on Monday is exhausting. Uh, we had a week to, to go between this time and it was awesome. Coaches. Uh, we got a lot of stuff done that we hadn't had time 'cause we're so immersed in the teams. Um, you know, and we were just better prepared, but we have a lot more to do. There again, you know, when, when he set that date for June 4th, um, hell, we probably could have planned for a year to try to get some of this stuff done.

00:24:50

That wasn't gonna happen. We needed to go testimonials. Um, things that they told us about their experience. Um, you know, one, one of the Scrum masters came in. She hadn't ever done it before. And, and she's, she's, uh, just extremely happy on how she's able to help her team execute ceremonies and, and really kind of help run the team. Um, one person said, we thought we knew what we were doing. And the Do Dojo helped us polish those behaviors. And again, those are kind of tweaking those levers and, and helping them mature. Um, we now function as a DevOps team. There was some other comments like, in, you know, what would you kind of say to other teams coming in? And it was, you know, be ready to, uh, learn. You know, you have to have the right mindset. And that's, again, part of the criteria. You have to see a sincerity about wanting to learn and, and wanting to get better.

00:25:49

What's next for us? So, like I said, Minneapolis, we just had a team, uh, start there. We need to build it out. Um, planning to have two to four teams if possible there in January. And, uh, you know, post Dojo support. If you think about it as we have more and more graduates, alumni, whatever you wanna call 'em, the snowball as it rolls down the hill, it gets bigger and bigger. How are we gonna handle that? Um, we need, we need to scale. We need to, you know, kind of bring on some more coaches and, and, and help with some of that stuff. Product owner specific training in our retro that we do with the teams as they're leaving. You know, that's something that while the Agile coaches are working with Scrum Masters and product owners, um, you know, we really need to hone in on that particular competency and help raise, uh, awareness to the, the business, uh, side executives as well.

00:26:41

Uh, Jean asked me, you know, if I could ma wave a magic wand, what would I do? I would increase our capacity within the physical locations by two to threefold. And the reason for that is I don't necessarily think that, you know, we need 18 teams at any given time. 'cause I have concerns about how we would drive consistency between the coaches. It's more about after the eight weeks, these teams really feel like they're having their umbilical cord cut and they, they get concerned like, Hey, we have you here, we're able to grab you. What do I do next week? And, and so it would be nice to kind of transition them through the Dojo and have a more graceful exit instead of just the, the chop at the end of the eight weeks. There's teams that need more help as they're leaving, and that's okay. We need to be able to address that. Um, I want to add more coaches. And the last one would be, uh, if, if I can ma wave the magic wand is to bring more teams through and bypass funding related constraints. <laugh> magic.

00:27:44

Okay. Where we can use your help. So, you know, just because I'm up here, um, I, I don't proclaim to be any kind of expert on this. Like I said, this is an experiment we continue to iterate. Um, for those of you who are already doing it, how have your org scaled up in the ability to, to coach and continue to deliver that consistency that we are looking for? Do you see anything missing from our program offering? Is there anything glaring to you? Um, and, and I'm all ears for any feedback that you guys have. You know, we, we all learn from each other. Um, it's, it's been fun to compare notes throughout the week, talking more with Ross's team at Verizon and, and hearing some of the things that they're going through. Again, it's all very therapeutic. We are hiring. So if anybody's interested in flight benefits, um, yeah, we are hiring, we, uh, you should see job postings. We're gonna have kind of a, a blitz, uh, out there of positions later this year and into January. So if you're interested in working for Delta, please reach out to me or watch delta jobs.net. And, uh, you know, I, I thank you guys very much for the time today and have a safe trip home.