00:00:04

It is an honor to be here. I'm Amy Bechtel. I have been practicing DevOps since 2013 and this is my fourth time presenting at the DOS Conference.

00:00:16

And I'm Abby Gray. I am a product manager at Capital One, uh, within our machine learning space where I am responsible for creating great customer experiences, but also ensuring the integrity of our customers identity and data. And I am by no means a DevOps expert.

00:00:36

I joined Capital One in December of 2015. Capital One has been on a technical transformation journey that I probably have followed over the years, and we are adopting agile DevOps in the cloud and we are actively embracing open source microservices, restful APIs, and are very serious about security. I joined our credit card technology group as a responsible for helping credit card technology advance their DevOps adoption and maturity. Capital One is a large and complex organization, and as you can see, uh, this is a very relative, very notional depiction of our organization. Uh, in order for me to find partners on the DevOps journey, I have to navigate a pretty complex and labyrinth of an environment. So today I'm gonna tell you a story and the first half of this story is a story about a journey that I took with a gentleman by the name of John Schmidt.

00:01:43

And John presented this talk with me in London, and he couldn't be here today and Abby has replaced him in that product manager role. But I'm amazed that I found John in the teams that Abby works with in such a complex organization. And so being hired to drive change and be a change agent every day I ask myself, how can I support adoption of DevOps in an organization of this size and this complexity? And I'm hoping the two stories that we tell you today, one about two web API teams and the journey we took them on using a Dojo, and then second, the machine learning and data science team that Abby is supporting right now and a year later can give the testimony to that sustained result.

00:02:30

This story begins in early 2017, and I was tasked with leading what was known as a DevOps acceleration service inside of Car Tech sponsored by our vice president. Our motto was No Fear Releases. Our mission was to go out and help teams adopt pipelines, continuous integration, continuous operations, and continuous deployment. I had a brand new team and I had no customers, a brand new service, brand new team. And when I was looking to find meaningful, purposeful work for that team, I reached out to my peers and my tech colleagues that were leading application, uh, leading these application teams. And I would pull up with them and say, Hey, I've got this great service. We can help you adopt pipelines. You can move faster. And I was surprised to be received with not. Now. We don't have time. Uh, you need to talk to our product owner.

00:03:26

And I heard, talk to our product owner, talk to our product owner a couple times, and it's dawned on me, well, of course, we are all in on DevOps. Our infrastructure and application engineers are all on the same team and sharing a backlog, the functional requirements, the non-functional requirements, and our product owners are prioritizing that backlog. And in order for me to get what we wanted to do into that backlog, I needed to start talking to the product owners. I needed to do something that did not come instinctively to me, and that was go sell to the business.

00:03:59

A little foreshadowing is I had a very narrow focus and I wasn't looking beyond my line of sight about who I needed to talk to and who I needed to be aware of and go outside the context in which I worked. I told my director at the time that I was experiencing some resistance and that I needed to go and engage with the product management and product ownership side. And he knew, knew of a product manager forum and he arranged for me to come speak at this forum. I had this awesome deck. I mean, it was got the 2016, it was 2017 at the time, mind you, so the 2016 DevOps report metrics, and I was gonna bring humane it and they could deliver features faster to the business. And I was, you know, doing this great talk at the end of my talk, I was expecting questions and hey, when can you start and sort of form a line out my virtual accelerator DevOps service door?

00:04:57

And instead I was received with glazed over looks and silence and just hoping to get some questions and some form of engagement. And I didn't except for one person, this gentleman by the name of John Schmidt, who was, oh, I, you know, I wanna talk to you. And I followed up with John and I wanted to believe that I said the magic words, that I had some script and I wanna set your expectations. I'm not gonna tell you what to say if you're looking to engage the business and get their buy-in. There are no magic words, there's no script. I didn't say anything in particular. What I learned was that John had visited the card, the Target Dojo in Minneapolis in the previous fall with several card tech leaders. And that he was aware of that model and he understood DevOps and had a foundational, uh, learning and understanding enough to know that he wanted to invest in it.

00:05:54

So I can't help but smirk as you tell that story because I sit in these product manager forums every month. And for the most of us, or for the most part, we don't know who Gene Kim is from Adam, and then the small majority of us who might know who he is, our knowledge typically ends with having read the Phoenix project.

00:06:13

When I started talking to John, he started explaining to me that he was working to implement some new leading edge technology, a real time streaming platform that was gonna have federated contributors contributing from all over the company. That they had a really lengthy delivery process. It was taking them weeks just to get out a single line of code or a change, and that he didn't feel his teams had sufficient DevOps skills despite being strong software engineers. And I was surprised to hear this, that he recognized these anti-patterns and that he was able to say, you know what? We have some issues and some constraints and limitations, and I wanna overcome those.

00:06:54

So this part like hurts me as a product manager for two reasons. First, is that going to have to talk to your investors and say, we cannot put out a change in a production. And anytime, uh, less than three weeks, I, and they're standing there with their checkbooks open and saying, what do you need? Do you need more people? Do you need more money? And we have to, I have to say, well, we didn't invest in a platform or, or a application that is extensible. Um, so adding additional hands to this isn't gonna help. But even greater than that and the way that the world is changing, not being able to make a change to meet our customer's needs in real time is just fully unacceptable.

00:07:38

So when I learned that John had done the Dojo and I had run a modified form of the Dojo for quite a while, he and I agreed that this is a model that we wanted to try and invest in with his teams. And I know you probably heard a lot about the Dojo this week, if you haven't. It is the Japanese place of the way. It is a whole team learning model. They bring their own intent and backlog into the Dojo. We pair them with masters or experts and they get a very safe place and environment in which they can learn and fail and quickly recover. I mentioned earlier the motto for the acceleration service that I was leading was No Fear Releases. And in Dojo our motto is No Fear Change. They're about six weeks in duration. And after running several dojos similar to the ones that we're about ready to describe, I will confidently say it is the fastest and most effective way to adapt a team to an engineering culture and accelerate expertise and adoption of tech technology. I have learned there's reluctance to go into this model that a lot of times product managers feel that they're slowing down intent and slowing down feature delivery. So John showed a lot of guts and courage and investing in this model, knowing that he was gonna slow down to speed up.

00:09:06

John formed a leadership team in his teams. There was two web API teams that he had and they had a tech lead, a delivery lead and a product owner. And they sat together the business and tech and worked side by side every day. I spent about three weeks scoping out a challenge. It was nine to 10 engineers. They had worked together for months. And this is important because this is a contrast to the data science team that we're gonna tell you about. They were co-located in our headquarters in McLean, strong software engineers, but limited or no DevOps skills. And we scoped out a six week challenge to implement CICD pipeline for each API on a federated code contribution going into contributing into GitHub. And these pipelines would both execute the same test code. I am a planner, I am type a, i.my i's and cross my T's. And I wanted this to go really well and be a hallmark service that we were gonna start providing. And, but there was a lot of uncertainty. I had to find a lot of experts. I had to have, uh, time things right and my anxiety was high.

00:10:14

You've heard a lot about psychological safety. This is what it looks like is recognizing that feeling and seeing how somebody might be feeling or what they're experiencing and validating it and saying, I can see you're anxious. That's okay. He, he, John wasn't trying to quell my anxiety or try to get it to go away. He just looked at me and said, yeah, it's gonna be messy and it'll be okay.

00:10:41

This is a relative journey map of the six weeks. And I can't get into the details of all the six weeks 'cause we don't have time. Um, but you'll notice the two dips and they're little J curves, if you're familiar with the J curve model. Uh, in the beginning they were really skeptical. They were, they felt that what they were doing was working and that they were able to put out quality code, but they didn't understand what the possibility was and they were resistant and uncomfortable. They were gonna feel exposed at what they didn't know a job, a person was gonna learn chef through pairing. And that very first week they hit a dip and they wanted, you can tell they wanted to go back to what they were doing. And it was their leadership. Their leadership stepped in and said, you need to trust Amy and her team. And it's a leap of faith, psychological safety. That's what it looks like. They kicked it in, they put in, in two weeks, they had a pipeline up and running. They were building and deploying to Deb. They felt really good. They felt like, oh, we've got this. We understand now. They wanted to go back to developing the features. Leadership had to step in again and said, no, we're gonna complete the challenge and we're gonna change the way we think in our practices. And they had to push through another pivot.

00:11:57

After six weeks I left the dojo, they went on to put in two or three more pipelines. They would not have been set up for long-term success if they didn't make this deep investment in evolving their culture in full team upskilling. I moved on and then John moved on to the machine learning in the data science space and he had the frame of reference in the experience. Now to recognize the antipater, it was a whole new ballgame. Again, a machine learning and data models, container orchestration platform they wanted to put in Kubernetes. It was a lengthy manual delivery process that took months, not just weeks. There were no DevOps skills. I have learned data scientists are not software engineers. Uh, they're very smart, but they were coming in with a very different experience level. John reached out to me and said, I wanna do another dojo.

00:12:55

Spent about three weeks. He had done the same thing. He had formed a leadership team with the product owner, the delivery lead, and the tech lead. It's about nine to 10 engineers. Again, these teams had never worked together before and they were in distributed locations. We multiple states, no DevOps skills with a tough challenge with a 10 week challenge. Another interesting challenge was that my team had never built and deployed data models. Uh, artificial intelligence and machine learning was just starting to get, uh, a, a hold at Capital One. We scoped out a challenge to upskill the data scientists to implement A-C-I-C-D pipeline for machine learning models to go into a Kubernetes orchestration in the cloud. Once again, my anxiety was high, the uncertainty was high, and John said, it's gonna be messy. This is the journey map of 10 weeks because it's, while it's the same pattern, it's a very different team that by the end of that first week data scientists, like they do prediction for a living and they couldn't predict whether they were gonna be able to learn these new skills.

00:14:06

And there was some tension and conflict and anxiety was high for this team. We had to bring in a coach and run a, what I call a mind mining session where we had to get what people were thinking and feeling out in the fears and address them head on. And that's what their leadership did. Their leadership said, you might fail. We might scratch this whole thing by the end of the 10 weeks, and that's okay. We will take the hit for that. And when that happened, they pivoted and then they doubled down. And within the first two weeks, they were building and deploying a data model into a Kubernetes cluster into uh, AWS dev environment. And then we hit the, do we hit the, what I call the mojo dojo dip in the holidays, uh, where everybody dispersed and they wanted to go back to developing the intent for the data models.

00:14:55

It once again, it took their leadership stepping in and saying, no, we are gonna push through and we're gonna complete the challenge and we're gonna get this in. This is an investment. So what took the first pivot to be successful with the API team was very different than what this team needed. And it's really important to put aside what you might think and be able to empathize with those teams in that psychologically safe environment and adopt and adapt to their values and a belief so that they can be successful and achieve the challenge. So I said there's two very similar patterns. You can see the emotional journey in the two js. Um, I like to look at the first one is the first, can we do this fear and faith dip? And then the second one is the, all right, I got it. Now I just wanna go back to what I was doing before.

00:15:50

The second one is the hardest one to push through. That's where you break those old patterns, thinking behaviors and push. And to achieve the challenge. There's been a lot of research on this. The six weeks minimum is by design. And I know target designs theirs as six weeks and now Verizon, it takes that long to instill new patterns and break old cycles. Really good video that reiterates and reinforces this is the backwards bicycle. If you haven't seen it, I highly recommend it, but it doesn't matter after the six weeks as much as it does to sustain that, you have to keep practicing and you have to have really strong leadership support. You've probably heard that a hundred times this week. That is the number one critical requirement in a transformation. And I'm so excited that Abby's here. She's working with these data scientists and has now since we left the dojo and can give the testimony because I've seen these two teams move on and what they've been capable of, and they are opened up that federated contributor model. They substantially reduced times and they upskilled their software engineers, they turn data scientists into software engineers. These guys are putting out Kubernetes clusters and owning and operating their own environment. It's amazing. So I am very passionate and convicted about this model, the Dojo model. And I can honestly stand behind that. It is the fastest way to produce results even when you have different types of teams such as the API team and the data science team.

00:17:31

And I'm just so thankful to be the beneficiary of the hard journey that Amy and John had to go through because what they did wasn't just a a 10 week dojo. They made a culture behavior change to the, to the point that like we are now a we we're not a siloed data science product tech. We are a single team with a single mission and we're having amazing return and amazing results. Um, personally, and this might sound like why wouldn't you do this? But it's gonna take more than just a couple non-functional stories thrown into your backlog. I go to my tech and data science folks before I go to my business folks to make sure that we're prioritizing the things we need to ensure that we're creating a product and a platform for long-term success rather than just continue to shove in in market value.

00:18:26

Um, and we've had such great return that I cannot share with all of you, but I did wanna highlight three of the points, um, that really resonate with the experience through the Dojo and, and things that are still there today. The first one being, um, we've invested in creating a platform that's extensible with reusable building blocks, a contributor model so that we can innovate faster. And by innovate, I mean we put things into production, get real time feedback and understand how our customers are responding to something versus having to spend all this upfront time building an underlying infrastructure before we can put something into production. And the second thing is something I like to call the oops tax. Um, I'm sure there is a technical term for this, um, but this is when I can get a group together, I can get some tech folks and data science folks.

00:19:19

We can put a model in production and see return. But eventually, if you do not invest in those sustainable and extensible platform, you're going to have a mountain of technical debt that will come back to bite you. And then finally, the flashiest thing and the coolest part of, or the most exciting part of me telling this story to my investors is the speed is real. We went from a mini month uncertain journey to put some of these machine learning models into production to a matter of hours and days. So now we're gonna switch gears, um, and talk about the product manager perspective on DevOps. Um, and I, I'm sure you guys work with product managers, but we talk a lot. Um, and part of me talking a lot is trying to tell compelling stories to my investors. So if you'll indulge me this afternoon, I'd like to try to piece together what a bear a picnic basket, an angry bag of money and a hero rowing down the river have in common.

00:20:23

Um, to do that though, wanna take a beat and remind every one of a somewhat ancient secret. And that if you want to influence an audience, you need to understand what motivates them. And within our motivational framework in general, we all of us wanna be a happy camper, which means we wanna avoid bears and find picnic baskets. And those picnic baskets might have intrinsic or extrinsic incentives, but understanding what those are for the people you're trying to influence is key for me. I would love to have a couple bottles of pinot noir and maybe the entire season of Grey's Anatomy and I would be a happy camper. Um, but something when I have a gr what I feel like is a great idea and I start to run with it and tell everyone about it, I forget the most important thing. And that our brains are disproportionately wired to care more about a bear than as many bottles of wine as you put in those picnic baskets. So until you address those people's concerns or there stakeholders concerns, you'll never be able to influence them.

00:21:29

Um, and I can't tell you what all your product managers bears are, but I can tell you what mine is. And for that I'd like to introduce you the angry bag of money and the angry bag of money appears when my investor, I've, I've succeeded in getting investment support, but I cannot re deliver return fast enough. And the angry bag of money sits there and looks at me and says, Hey, I did my job. Why can't you get in market value faster? And this might be because we have constraints or competitors come into the environment, but this happens all the time. So it's really important that you prioritize or you set expectations and that we might have to do upfront investment, like Amy said, with the dojo to go slow so we can go fast in the long term.

00:22:21

Um, so second metaphor, second story is probably a little bit risky to attempt in a room full of engineering backgrounds. But for me it's a simple law of physics that if we do not pair safety with sp sorry, safety with the speed we are trying to achieve, we're inevitably going to lead to a crash. And this kind of brings us back to the oops tax of, yeah, I can get in market value, but if I'm not pairing that with the foundational things that DevOps brings to the table, then I'm inevitably gonna crash inevitably have an oops and not be able to continue because of a mountain of technical debt that I can never get rid of.

00:23:05

Okay, so talked a little bit about two different metaphors. Um, and I wanna try to bump this all the way up to an allegory where we have a hero rowing down the river and he's in a race, he's a pretty strong rower, he's feeling pretty confident and he is really focused at the task in hand. And he's going past Amy and he's waving and maybe looking around and teasing her a little bit, standing up, taking in the beautiful view. But there's a huge problem. And the first one is that the downstream speed of the current is faster than the rower is rowing. And not only that, there is a giant waterfall at the end of this mountain or at the end of this river because he's so focused on the race. If he doesn't look up, if he doesn't look around him, he is going to go over the waterfall.

00:23:56

And this brings it all back together. Um, because if we are not learning faster than the world around us is changing, then we're inevitably going to go over the waterfall. And as a product manager, I care about investing in software that is going to create, but also to maintain a competitive advantage. And the only way to do that, in my opinion, is to outlearn people. It's not to out to deliver. So you have to look around you because guess what, Amy just got a ro, uh, got a motorboat and maybe she got an airplane and the rules of the games have changed. So you cannot just focus on the the race at hand. You have to be aware of the things that are going on around you. Um, so for me, the dojo gives me the speed that I need to get off the ground, but it's paired with the DevOps safety that allows me to continue to be successful so that the next time when I move on to the next role and start to see the same signs that John and Amy have seen, there are two examples previously I know who I'm gonna call to have another Dojo.

00:25:09

Thank you Abby. And I'm gonna be in a submarine. You're not gonna see me. <laugh>, <laugh>. If I could go back and talk to the early 2017 Amy who was getting ready to start and lead this acceleration service, I would tell her these nine to 10 things. I would tell her to look for the innovative forward-thinking product manager, not the tech leads. I would engage both product management and tech and understand what motivates them and who they are and empathize with them. I would frame my arguments around outcomes and not velocity or, uh, story points. I would not, I would also look at the model for the product leadership team and form one of those and recommend that I would hands down lever the leverage, the Dojo immersive learning model to accelerate that change and to jumpstart a transformation for a team, I would have product leadership motivate and support that team through those dips. Leadership is so critical in this journey. And I would avoid that narrow perspec perspective. And while I was in my rowboat, maybe rowing faster than the teams around me, I would also be looking at that speedboat and the potential submarine. And then I would celebrate the success, success that we are achieving as a team. And the most important of all, I would continue to build that relationship with my business partner.

00:26:40

What do we need help with? Looking for effective models to support culture change and not just technology changes other than a Dojo more suggestions on how to influence the business. So they choose DevOps first and go pipeline first, and more suggestions on how to influence engineers to want to learn as a team and commit to continuous improvement in learning. So thank you. It was an honor to be able to present our story. And good luck with yours. Thank you.