DevOps Beyond the Tools (London 2020)

When IT is transformed by technical enthusiasts, the amount of time discussing tools and technology is immense. We love to elaborate which integration, product or concept would solve something, or even “everything”. Often though, this is at the cost of the other two pillars of a transformation besides Tools – People and Processes. Andrea and Duncan will discuss what needs to happen to get beyond talking about the tools aspects and to start putting focus on process and people.

2020plenarylondon
AH

Andrea Hausmann

Head of Program Strategy & Engagement, Credit Suisse

DL

Duncan Lawie

Director, DevOps and Development Practices, Credit Suisse

TRANSCRIPT

00:00:09

Andrea Hausmann is head of program strategy and engagement for credit Suisse. And she is presenting with Duncan Lowie head of strategy and governance. They're both part of the DevOps and development practices group responsible for helping elevate the state of engineering across the enterprise. They are talking about their surprising revelations as they decided where to focus their efforts and the refusal to be trapped into arguments about whether this tool is better than that tool, or even about tool standardization strategies. They reached some startling conclusions, which I think will be relevant to everyone in this community. Personally, I'm so delighted by where this journey took them. Please welcome Andrea and Dunkin.

00:00:51

Hello, to provide a little context for our conversation today, I'd like to give you a brief introduction to credit Swiss. The bank was founded over 160 years ago and has about 48,000 employees. Today. We have one and a half trillion in assets, under management in Swiss francs, as our headquarters is in Zurich in Switzerland. In, as you can see in that, in the picture here, we serve our customers. However, around the world through three feet, regionally focused divisions, the Swiss universal bank, international wealth management, and the Asia Pacific division, along with specialized investment banking capabilities from global market and investment banking and capital markets. And of course behind that, we have our shared services divisions, which includes our central it organization. Andrea Houseman and I are both Duncan. Louie are both part of that central group with Andrea covering program strategy and engagement within the dev ops and development practices group, which is part of the it strategy and architecture organization. Whilst I work in the same area, covering a broader space of strategy and governance, Andrea, over to you.

00:01:59

Thank you. So this presentation is about patterns, mostly cultural reoccurring over and over again, irrespectively of the job or the industry we talk about. So we want to highlight importancy of critical thinking, as well as keeping our minds sharp and thoughts that we always be aware of such behavioral challenges. There is not the solution to any of that. Did we at credit Suisse have tried to come up with a few best practices we'd like to share with you today to overcome such, um, some of them, but unfortunately we don't have solutions to all of that yet. So if you have something to share with us afterwards, you're happy to learn. But I think what is important behind that is that we know that these challenges are existing rather than really having the solution to everything. So storyline today of this presentation is rather, um, is amongst my rather short career so far. And there's actually two things, both stunk. And I have worked in, in a pulse. We both, both enjoyed very much. So one of this is dev ops obviously, and your thing is working at the bar.

00:03:04

So very beginning of my professional career, I was working in Surmont, which is a beautiful mountain village in, in Switzerland. You can actually see the mother, the horn, the pixel on the picture. And we were at the time at that time, re-establishing the bar from scratch. So we had to order everything from beer to wine to spirits. We also bought utensils like what cash system do we use? What losses do we buy? But also that at some point, what course group, for example, do we need to use as a more keeper team? And suddenly we found ourself in a discussion of bar keepers, discussing which corkscrew is the best and easiest to, to handle. And everyone had their advantages and disadvantages in terms of stability, price, sustainability, ease of usage, probably how long they're going to last. Um, and there's, there was no, we couldn't come to a very easy discussion off their own.

00:03:56

But if you think about what really is the value, which derives from a core screw, it is about opening a wine bottle, pouring it into the glass of wine, bring to the customer that should optimally be quick, and there should be no leftover from the cork in that glass and very good wine. Nowadays don't even have court sometime anymore and many people. And because we were not a sole wine bar have ordered beer or 10 tonic. So this discussion is probably not that relevant in the end. And the ultimate goal we must keep in mind is to give the best service possible to the customer while we enjoy ourselves fast forward a few years, I was graduating from university in business mathematics. I joined numbers. I love to calculate those big complex equations manually by hand. Unfortunately, I was born a few years too late and this kind of job doesn't exist anymore.

00:04:51

So I got myself a job as close to that as possible, doing the same thing, but with the computer that was hired as a data analyst in credit Swiss, we've been analyzing it operational data. So for example, defining making prediction, how healthy a server is and see when they're probably going to be the next outage. Once I had my mathematical model verified, we had to get that to the customer. Obviously my mathematical proof will not help the customer understand very much what I'm trying to say in that proof, nor is it very user friendly. So all of a sudden we had a discussion in the data analyst team. How do we bring that now to the customer and discussion started arising from what are we using Tableau or power BI, or should we even build our own glee? And they're all have their advantages or disadvantages because I couldn't decide on, um, what do we go with Tableau or power BI.

00:05:48

We decided to go with both. So me and my colleagues started building a parallel. I went for the Tableau solution and he went with power BI and we both been developing. And this process was, was actually very fun because we've been, um, challenging each other and seeing how we progress. And we're seeing disadvantages and advantages of each of the tools. But what it ultimately meant is that it definitely delayed the shipping of my valuable information from mathematical proof to the customer. And it definitely costs way too much because we had two people building on two dashboards, even though just one was needed.

00:06:25

Um, and don't get me wrong. It is important to think beforehand about Vox, what functionality does actually my dashboard need. And probably also think from a user perspective, not just from me as a developer perspective, but it is rather about what functionality does the dashboard need to have then which tool do we ultimately use. So again, again, the angle is to provide the best service possible to the customer that in this case, he can make informed decisions. Um, and we should enjoy while doing that. And we shouldn't delay the slippage of the dashboard. So this has been now, um, two examples about, um, such behavior, but not in terms of dev ops, actually. So dev ops was my last big change in Korea direction. And this is actually where Dunkin I'm at. When I joined the team, Duncan was at that time co running the enterprise wide dev ops working group that their ops enterprise working group and correct Suisse was a group of senior leaders thinking about the future of credit Swiss in terms of DevOps, as it is a enterprise wide, a divisional thing, prostitution thing, um, each division were able to send their delegate to this working group.

00:07:35

So each of them have sent their probably most senior or more technology averse person to that working group. And guess what happens if we'll talk more than 10:00 AM such leaders and technologists into one room, the discussion about tools get even larger, probably more aggressive and definitely more political.

00:07:55

As Andrea said, when we met, I was already co-running the enterprise DevOps working group, but that was a journey that was a long way into my career journey. With over 20 years of it experience, I was quite confident that I knew where my career was going. I had a great deal of skills, and it seemed to me that when the term dev ops first came up, it was yet another one of those terms that consultants want to whitewash the world with. And it doesn't make a real difference at all. But when I spend a bit more time looking at the topic and trying to understand it, I realized there might be real opportunities for me in my career for better success in the teams I work with and possibly even a bit more happiness for our organization. And I think it's important for us to remember where we have come from as senior leaders and to have empathy for people who have not yet made that much of the journey and to think about what will actually bring them with us towards the things that we think dev ops can really do for us.

00:08:50

As Andrea said, when we got together, the easiest conversation for us to have was about technology, which is better Jenkins or team city, what's the right way to use JIRA. And of course, these are the kinds of lively conversations that we'd love to have. There was plenty of heat in the room, but sometimes perhaps not so much light, but it really gave us a feeling that we were talking about the important things in the world, but it didn't solve all of our problems. And so we found that we also needed to talk about processes and processes in an organization like credit Suisse, follow Conway's law, unsurprisingly complex organizations, multiple divisions. We found that our processes were also shaped like that and that the control handed from one part to the next, across the organization. And of course with the older processes, there were things that actually reflected previous iterations of our organization and were even more difficult to understand.

00:09:46

So we did work in within individual visions around this, primarily around value stream mapping. But one of the other challenges we found here was that really people's egos are built into these things. I mean, I had worked on how we did our database provisioning process. It was very important to me that we had learned there. So we had delivered this, we had built something that was better than ever existed before, but actually kind of coming back to that and taking my own ego out of it and looking at what we could do next was a significant challenge. And we need to recognize again, that that people have built their careers and feel that that is what they need to have and to show off forever. So there was not a lot of low-hanging fruit in that space. So we also talked about people and in the people's fear, you know, the group in, in Asia Pacific who had first started working on dev ops, had made organizational changes. They had put people into dev and ops focus, grooms groups. They had also focused on how they manage their hiring process and what kind of things should be, how we, people should be rewarded in a technology organization. But that was not necessarily simple. It wasn't whilst the group of credit Swiss as a whole understood. We wanted to make these changes. The process of making those changes, the effort required was, was slow, which meant that there was not a lot of quick wins available there, either

00:11:12

Going back to the statement, I just said before that the discussion about tools got larger, more aggressive and definitely more political. The next few things I'm going to mention now are things you probably have experienced your colleagues act against, and probably even yourself have had this behavior in Nepal. So these are typically human flaws and probably it's not even floss, but just how we as human functioning in order to survive. So for example, people are getting motivated by different reasons. Some people are getting motivated by cognitive reasons. Some people are getting motivated by altruistic reasons. And some people are getting made motivated by power. If someone is getting motivated by power, that they tend to make sometimes irrational decisions just because they know this decision will probably help them grow their personal impact they're having, or it can get even worse. People might tend, um, for a solution or making a decision because they know this decision will not shrink their power, their power impact area.

00:12:13

I'd like to call the garden. People are liking to look after their garden and this is a very human behavior. Um, and those people normally don't do that because they are they're evil or anything like that. It's really just, that's how humanity in the end works. And it's a very thin line for the companies and for decisions to make sure we take care of that. This is a human behavior, and we cannot, um, well, must take care of people behind that behave and decisions, but also that we try to minimize those personal biases as much as possible in companies. And also I think if I, if I think about how many people I personally know who would put the company success before their own success, I think there wouldn't be too many.

00:12:56

Um, another big pillar in complex and big organizations are politics and politics. You might again, um, play into someone's card because you know, this person will repay you at some point or this person might even help you progress on your career. And this behavior is very toxic. This very, not what we should have in companies, but such a transformation can take years. And depending on what kind of senior leadership they're having will not ever, ever even happen. And you, you attending today at this conference, you are the middle management and senior management who can actually change that kind of behavior in your organization.

00:13:38

And one of the critical things for us to do to make those changes is to move away from the hippo, the highest paid person's opinion and move towards being more of a data driven organization. Now that might sound like a slightly tired term two or three years later, but it was just what we needed to think about at the time. And so we looked at what data was available and what we could do with it. And of course, to begin with the data that was available to us was the information that had been important to us in the past. And quite often, these were things that, that either we had chosen those measures because they made us look good, or we had worked out how to make ourselves look good against those measures. So they're not necessarily vanity metrics, but they are the kinds of things that help to make sure that you get a nice green chart on your, on your own list.

00:14:24

And therefore there was some resistance to changing those measures. And as we move forward, one of the key things that made a difference to us was that we needed to see how we could use the light, where it was a little bit like looking for your keys under the streetlight, not necessarily because you lost them there, but because that is where the most light is. And so the next thing for us to do was to try and spread that light out. And to be honest, a critical thing, there was actually having enough in our organization having read this book, once enough people had read accelerate, we could start talking about outcome measures. We could start talking about a common industry span and set of metrics. And we felt that we were all talking about the same thing, that we had a common understanding of what we were talking about.

00:15:09

But one, even once you've agreed on those metrics, actually measuring them can still be a challenge. Lead time to change is quite easy to define from the point at commit to the point that something is actually released into production. But if you don't have a single chain of control through that process, it can be very difficult to actually measure that amount of time. And indeed some of our more advanced groups were using pipelines and processes that let them actually do that end to end measure. But if we just measured that those teams, we would be actually losing a lot of our opportunity for growth, our proof of improvement of moving towards pipelines. Another example from the other side is that if we look at change related failure, some of our divisions are very latency sensitive. They care a lot about small issues. They will shout as soon as they see something.

00:15:57

If you compare that with another group who are more interested in long-term measures and have less sensitivity to moment to moment changes, they may appear to have a long, a lot smaller change related failure rate. And that brings us to a place where the organization is afraid of the comparison between divisions, because these guys look like they're doing a lot worse than those guys here. Again, we need to make sure that both are on the trajectory of improvement rather than trying to make unreasonable comparisons between divisions. So all this combination meant that we felt we had a compass, but we didn't know if it was a perfect compass. Clearly the important thing here is that we have a rough idea of where north was, and it was important to start moving, to look at the fact that it might not be compass north. It might not be true north, but we could get towards our targets.

00:16:46

We could start to make real progress rather than waiting for a perfect compass. And as we get closer, we can improve the quality of our measurements and narrow in on our focus. And indeed that is the work we're doing now is to continue to improve the quality of our measures. Another thing that we wanted to look at was staff satisfaction. My favorite definition of dev ops is still the one-liner from John smart, better value, sooner, safer, happier. And we're talking about a lot of the aspects of this through the conference, but talking about happiness in a big corporation can be difficult. People have different ideas of what happiness is. People have concerns about actually asking people in their own jobs. Are you happy? And indeed, but, but my feeling is the best way to do this is to actually ask people. But we had, I had created a challenge there myself because when we first started talking about DevOps, we didn't have enough measures.

00:17:43

And we sent out surveys. We asked people how dev ops are you? And it turns out that on a sunny Friday afternoon, people felt that they are more DevOps than on a wet Monday morning. The critical thing here is that we should have been using telemetry instrumentation to measure those things, but some things are about people and the easiest way to find out what a person thinks I believe is to ask that person. So from there, I ended up having to talk to associates across the industry to understand what they were finding when they attempted to do this kind of thing. One organization, a leader of a thousand people said, I know, I know what my people think, and he didn't need to actually run a survey. Well, potentially he's a genius potentially he's missing fault in either case, surely you can get good for quality of data, but even to confirm his hypothesis by running a survey, another example was an organization where the leadership team, frankly, were afraid to ask their staff, how do you feel?

00:18:42

Because they felt that their staff were not happy and they didn't want to be put into a situation where they would have to ask them and then do something about it. Bringing that back into credit Swiss. We thought about the way that we wrote our satisfaction survey and what the questions should be, so that we could work out what we would do with the answers. And we used the results of our satisfaction survey to prioritize our ongoing activities. This made a significant difference to the quality of our outcomes. And, but of course, we also made sure that there was room for comments, for individual experience, experiences, and responses that could guide us about the things that we didn't already know. This fund comes under the term of engagement through dialogue of working across the organization and building our own shared vision

00:19:34

Through such a shared vision a few years back, um, a common, a tool chain and even dev ops end-to-end got developed in credit Swiss. So this tool chain really enables everything from collaboration tool to code reviews, search functionalities, pipelines, build, test, deploy security analytics for technical depth. It's a huge thing. So Duncan, I actually joined that team a few months back. So when it was originally built this, this tool chain, it was living and, uh, in a business division and serving this business division and suddenly it was realized how great and important that will be for the whole organization. So it got moved into the central it function as is living now in central. It function has to serve all the functions and all the other businesses with that it's suddenly has more than 20,000 it employees to serve with that. You definitely have 20,000 different opinions about what you should do next.

00:20:31

So definitely there was, um, as soon as this was moved into a strategic place, this discussion got cute because everybody had the feeling that their own very own division probably would need a tiny bit differently that they can leverage what they need to do the best afterwards. If you think about, we really have different balance sheets for those divisions. So they ultimately have different ankles in mind. So this wasn't easy and Sundar were coming competing solution challenging, um, trying to challenge our approach of the central tool chain. And, uh, don't get me wrong here. We don't want to, we think challenging the status quo is very important and we think, and ownership is the right thing to do. We must do it in a sustainable and reusable way. And as money doesn't grow on trees, not even in banks and Northwell resources. And we definitely don't want to build endless complexity in our, into our ecosystem. We must be careful what we do and have clear eye on that. And if you think about it in a game theory terms, if the plan is not that one player gets from it, everything, and as much as possible, but that everyone, all the players profit from it as much as possible.

00:21:44

So this central tool chain is now a huge success in credit Suisse. We're having now even business user on our central tool chain using a few parts of the tool chain. And with that, we have all of them 40,000 users, and more than 38,000 daily users, surely there will always be a few complaints and few concerns from, from users. What could be changed whatsoever, but ultimately two things helped us get getting the tool chain into the state. Everybody can accept that and live with that and really support it. One thing is we are 30% of finance externally by our customers while internal customers in credit Suisse. And that helps us. We always say customer focus. The second thing is we ha we are living actually in the sourcing approach. That means that everybody in credit Swiss can put their, um, development and extensions into our tool, into the wish, as long as they don't break anything or harm anything else, which is already built. They're free to put any extensions into our cram, into our tool, and if it helps them to grow something better.

00:22:48

And so where do we get to here? My favorite term for this is to talk about balance and perhaps balance is just another way of saying systems thinking that first way of dev ops, but there's a balance that we need to find between the individual and the team between the team and our business customer. And it starts with the individual. One of the responses we got from the feedback survey was to was for people who were concerned that they were being asked to do too much in too many different dimensions and to somehow abandon that deep skill that they had now, that was definitely an, is definitely not our intent. People like me. Who've got 20 years of experience in a space or whatever the necessary time is, should be able to concentrate on where they are technically. Excellent. And one of the things that we do here is we've started doing practitioner to practitioner sessions so that those people can share their expertise and we can build the knowledge of the whole organization, but equally people need to be, to think about what the other, what the rest of their team is doing and build out a broad understanding of the rest of the organization.

00:23:52

And we're doing that primarily through communities of practice, which will focus on broad topics like dev ops and agile and so forth. And therefore they can learn a little bit about what other people do so equally, if you're early in your career, perhaps what you spend more time doing is building is looking at that broad opportunity and thinking about where exactly you want to actually root down into deep skills. And the result is you end up with the T-shaped skillset that people talk about, talk about, and the agencies of those teas join up to give you a team that can work together across the topics that we want to cover. And that in the end gives you that dev ops result. You keep your team in synchronization, you have your balance between the development goals and the operational goals. Again, those Dora metrics have two measures that focus on the kinds of things that developers primarily care about, speed of delivery, and two measures that operational people primarily care about the stability of your operational environment and balancing those goals lets the whole team improve.

00:24:54

And also it helps them with the empathy for each other. And there are other separate goals that they might have otherwise beyond that, of course the purpose of it in a bank is actually to deliver to our customers, whether that's the end customer in Zurich, in Switzerland, for example, we have a major high street banking operation globally. Perhaps we're focusing more on relationship managers who are helping their customers, who are helping with investment processes, which, which whoever is actually the consumer of our technology. We need to think about what we're delivering to them as the primary focus, but still there's a balance needed. If you focus only on delivery to the customer, then you're tempted to put in hacks and shortcuts, create manual work, to get work done, to get deliveries done faster. But doing that puts you in a place where you're creating an unsustainable situation for yourself. And the team needs to spend some time on self-care on doing the things that make sure that we succeed and continue to succeed as a team. Otherwise he lead to burnout and eventually the failure to deliver anything to production. So focusing on that balance on reducing manual processes and toil also helps with the customer and continuing delivery of success to the organization.

00:26:11

So last but not least, I would like to talk about a very delicate topic. And as I mentioned before, the ultimate value we create as it to the banking is either solving our end customer or for our shipping good code quality to our front facing employees. So the pure of other creation lies with the engineer. And if you think about, um, the core versus context framework, the core actually is the engineer and the context is everything else. So we must make sure that the core part gets bigger and the context stays a smaller as it necessarily needs to be what we did in, in our division to make sure we are following this approach, which is not an easy one. Um, is we, we put a different mechanism into hiring process. So if a hiring manager would like to hire an engineer, it is quite easy.

00:27:00

There's a shortcut through it. There's not much to do not much. It's just defy and they will get their spot approved and can put the offer to the market. If you'd like to hire someone else than an engineer, some example, someone like me, the hiring process is tiring. So there are many obstacles we built in on purpose to make this process painful. So the hiring manager really think about is this what we want to do? Is this the way forward? But surely engineers alone will not rule the world. So there is like with everything in the world and need for diversity. And as we discussed before this pattern with a discussion about tools, if you plug in 10 engineers into one room, we can guess what happens eventually. So there is this need for diversity that, for example, so many in the room, probably at some point will point back to the original problem saying like, this is probably first, we need to discuss about the process behind it rather than how do we resolve that with which tool in the end. And ultimately, always, we need to think about the ankle. And if we think about this conference, they're holding here at the moment, for example, um, the devil's transformation, which we're all very much interested in. We do not transform because it's funny or because it grows, it grows our impact. We do transform because we ultimately want to reach the goal. And the goal is to be generating the value better, sooner, safer, and happier.

00:28:23

So are we achieving that? Yes, I believe we are. We are having an impact. I mean, some of these are ACOs. These comments include our customers across the organization and other parts of it, but I'd like to focus on the one from Laura. Barrowman twice mentioned in the FinTech news list of the most influential women in finance in Europe, she's clearly influential in credit Swiss as the group CIO, she has 12,000 staff. So she, what she says matters. And on this topic, she said the dev ops program has served ingrain a common set of goals across credit, Swiss it while igniting a transformational shift in our organizational culture and mindset. So we are on the journey, but we're far from finished. We have more mountains to climb. The questions we'd like to thoughts from you on, particularly at this point are how do you focus on core engineering activities, over context? How do you talk about happiness in your organization and how do you test whether the message is reaching through the whole company? Because the more people we have together with us on the journey, the easier it is to reach our goals. Thank you.