00:00:04

Hi, we are here from JP Morgan Chase. Uh, so my name is Kenneth McLeish and I run Equities Technology. And this is, uh, my colleague Danny Myers, who runs Equities Production Management and is also runs the DevOps community of practice in London, or DevOps Danny as we call them. Um, so lemme give you a little back bit of background about JP Morgan Chase. Hopefully, you know, you know us. Uh, we are one of the biggest financial institutions in the world. Um, we manage assets of $2.6 trillion, and we've got a history going back over 200 years. We operate in over a hundred countries and we employ over a quarter of a million people. So it's a massive organization that ranges from, um, everything from retail banking, ATMs, big mainframes, um, all the way through corporate finance, uh, investment banking, asset management, and things like high frequency trading and, uh, a wide range of technologies there.

00:01:02

And in doing that, we have over 50,000 technology employees and spend $10 billion a year on technology. So looking at in that kind of dimension, um, we are in, in many ways a technology company and we've been doing technology for a very long time, and it has always been core to what we do. Um, so if you think, even think back to the eighties, we were electronically trading even before there was the internet, but in many of the ways of building software at that time were not modern. Um, and there are new techniques that have been learned, and it's really come as an organization that from the top of the organization, people have said, there must be a way that we can be developing software better. We can learn from others. Um, we can elevate the, the principles that we apply in building software, uh, in developing software to be more modern. So right at the top of house, there was a framework that had been created to try and help facilitate, uh, the adoption of new modern development techniques and DevOps. And that comes through things like learning and development, um, new career paths for people, um, bringing in new tools, um, looking at additional metrics that we can track the capabilities of different teams. And then, uh, creating communities of practice where we can share knowledge and try and encourage others, uh, across the various groups to do, to do more.

00:02:31

We'd now like to take you on our team's journey and highlight the impact and contribution that our team has had within the organization. We work with an asset management and we're focused within the equities trading technology team. It is our systems that generate in 2017 $1 trillion worth of trading activity, which is an equivalent of $4 billion daily volume. We trade across six trading desks, which is across, um, six trading desks across 70 financial global markets. So we're operating a very significant scale. Our journey started in 2013 where we have many monolithic and legacy systems that we are operating in equities trading alone. We have 13 systems that were mostly a combination of regional trading systems and reporting systems releases at the time were very infrequent. We actually had a very much of a structured plan, build and operate model that we were working within. And our, we, we had to have lots of handoffs between the different teams as part of that cycle. Releases didn't come without issue either. In fact, they were, they were troublesome, dramatic events where they typically caused incidents. As you can see, there were 10 that year. And as a result, we had incidents from releases and system outages, so no one really looked forward to them. Our code coverage at the time was only around 36% across our system, so there was a lot of manual testing involved. It also took us around a week to ship our code to production. Excuse me.

00:04:16

Our technology team was, was very much split at the time. So we had a separate dev team and a separate ops team as part of the segregation of duties model that was linked back to the plan build, operate model. In the ops team, we had eight people. The development team was made up of 36 people. Now that wasn't solely developers, in fact, it was only 50% developers. The rest were manual testers, um, business analysts and project managers. And what we'll do now is we'll now take you on the journey in more detail and cover off some of the challenges and principles that we applied.

00:04:53

So I wanna start talking about some of the principles. So the, the first challenge that we faced was how to get started. So given that we had such a critical business that we were supporting and had such an archaic platform that we were looking after, then the real challenge was the developers felt that they were just desperately trying to keep the car on the track. Um, it felt it had some dents and, and, and bashes. Um, and yet what we were wanting to do was completely modernize, uh, the vehicle that we were operating while it was still, while it was still driving. So that was a huge big challenge. So one of the first principles that we then looked at for the team, and, and this is something that some of you may have come across, but the sphere of influence and control. So the principle states that there are some things that are in your sphere of control.

00:05:38

So for instance, what you're wearing that day, um, you completely decide what what you can wear. Um, and there are some things that you can influence and there are many things that are in your sphere of concern, but you have absolutely no control or no influence over. And often you'll find people will spend a lot of time, um, worrying about trying to change the things that are in their sphere of, uh, they're outside of their sphere of control or influence, and they end up getting frustrated. They end up becoming the kind of complainers and people start listening to them less, and ultimately their sphere of influence and control starts to shrink. But for those that focus on the things that are within their control and being good at that, then over time the things that they get given starts to grow. And there's fear of control, uh, and influence starts to grow as well.

00:06:26

So looking at that principle, and one of the things that, uh, we dealt with was, uh, our change process. So the change process that we had was very onerous. So you can see on the screen here, um, there were multiple parts that developers had to fill in before something went into release. Um, it was time consuming, it needed lots of different teams to get involved. Um, and it was something that the developers wanted nothing more than to get rid of it. But ultimately, given, you know, we're a highly regulated business, that this is something that was outside of our control to get rid of. Um, so the key part was then reminding the team, okay, what do we have that we do control that we can apply to make this better? So the first thing that we looked at was actually tools like these within siloed organizations can end up becoming the communication mechanism between teams.

00:07:21

And that is massively inefficient. So you'd have developers write up all the details of about their software, and then they'd click submit, which would go into a queue, which would end up going to another team that would then review that information, come back with some questions, hit reject that would bounce back. And so that was one of the killer parts that was slowing the whole process down, but that was in our control to change. So we really changed how we were engaging as teams. So first of all, the, we would communicate more often and earlier in the process, so we would start thrashing out some of those issues earlier on, and even to the point where we could actually start sitting teams together. So sitting support and development side by side had a massive impact in making this more efficient. And so this really started to transform, uh, even within the piece that we control how efficient this was for us. And then over time, as we started gaining momentum and getting faster in our changes, then we were also invited by other teams to then, uh, help them. And then that ultimately led to us being involved in the, with the change group in terms of choosing a new tool that will ultimately completely streamline this process and automate it.

00:08:35

Our second challenge was with tooling. So as our two teams moved closer together, we realized that there were a number of tools that we were using that were producing the same result. So we had to assess those tools and consolidate them to choose the, the best tool that would fit the job to be perfect. So we af after looking at the tooling aspect, we were able to reduce the tools and make sure that we had the right tool to do the job. Now, by doing that, we could then become much more streamlined in what we were doing and how we were operating. So the principle that we applied that we were looking at was called nudge theory. So nudge theory is, um, a principle about how you can encourage behavior by making it simple to follow and, and nudging people in the right direction. So from the image you can see Richard Thaler, who entitled the book called Nudge, and he came up with a number of behavioral science concepts that can encourage people to choose the right behavior.

00:09:37

So from the quote here, this is take actually this, this quote is taken from the UK behavioral sciences team who conducted a lot of investigation into how you can nudge and encourage behavior through government. So they coined an acronym called East. So you can encourage behavior through making it easy. So if you remove a hassle factor from that through making attractive if you incentivize it or have rewards associated with it, making it social through social norms, or if other people already adopting it and making it timely by having the most appropriate time when people are receptive to that. Now, there is a funny example of nudge theory that is quite used in that is quite useful in everyday life. And I'll just like to draw upon that. So this is, this is a menger ial. So so many of you will recognize this, uh, and nudge nudge theory has been applied here. So there was an example, um, there was a study carried out in, in Skippo airport in Amsterdam, and they were trying to identify how they could reduce spillage in the men's rhinos <laugh>. And to do this, they conducted lots of research because what they had to do was find the optimal place for where a stream could enter the urinal and produce the least splashback

00:10:54

<laugh>.

00:10:56

And once they had conducted all these rigorous tests to do that, they had identified that exact spot. And I haven't actually got it on here, this is actually a football and a goal, but it's still a target. But on, on this example, they had stuck a sticker of a fly and that became the optimal spot. And what it did was it, it decreased the cleanup time in the toilets, in the men's toilets masse, um, by a whopping 80%. Now, as, as I'm sure you can, all, you know, most of you can appreciate us men, you know, are simple creatures,

00:11:35

<laugh>.

00:11:37

So if, you know, if we see a fly or, or a bull and a goal, you know, some type of target that we can aim at, then we're most likely to do that

00:11:46

<laugh>.

00:11:48

Now obviously this is, this is, you know, not a talk about toilets and, and, and anything else. We're talking about DevOps and, and, and, and nudge theory specifically. So how, you know, in JP Morgan did we apply nudge theory? So a tool called app fit was developed, and the purpose of this tool is to assess each applications software development lifecycle. And that's every application within the bank. So, you know, we have 50,000 technologists within the bank. And this, um, application is used to test, um, every application's SDLC best practices. So there are various measures that, that it looks at, um, that, you know, cover from, you know, how often you're committing, how often you're doing automated te the, the volume of automated testing that you're doing, is it above a certain percentage and how often you're deploying. Now the important thing here is to note there are pictures of robots on the screen within an app fit.

00:12:44

And to determine how healthy your application is, you get, um, an image of a robot to term. Um, you get an image of a robot graded against your application. So if you're ticking none or some of the boxes, you may have the broken robot. No one wants a broken robot. The power of the image is so important because us previously we would've had maybe a scorecard that would've just been a checklist of how well we're doing, but it's the picture of the robot that is making the most difference here. So the broken robot, you're not really doing too well, the awkward looking robot, you're doing all right, but you know, it's, it still looks like you're struggling, like you're a bit, you're, it's almost like that robot's in a straight jacket. Ultimately what everyone wants to have is a picture of the happy dancing robot that's doing the jig, because this is showing that it's following, that your application is following the best practice and is recognized as, as leading the way in terms of how you're applying your software development lifecycle processes.

00:13:49

So the next challenge we faced was one of technical debt. So every, every system has has technical debt. We had lots. So some of our systems were 20 years old. Some of it was written in Visual Basic. Um, so it wasn't designed to, uh, built for testing, built for, you know, frequent releases. And so for, again, for the development teams, that really felt like a mountain to climb, um, to ultimately replace many of those systems, many of those components and start to tackle that technical debt problem. So the principle we looked at for this, this piece was one around marginal gains. So marginal gains was a, a, a term that was coined by, uh, David Brailsford. Now David Brailsford, he is the British cycling coach, um, and the, the coach of Team Sky. And so what he said, this is a quote for him. Um, so you can achieve optimal performance by the aggregation of marginal gains.

00:14:45

And so this is what he started to apply when he became the coach in 2002. And at that point, uh, British cycling was, was terrible. They'd won one medal in 78 years. Um, so it wasn't doing well, but he was saying that if you can be 1% better at everything you do that a aggregates together to make you much, much better overall. And so some of the, there's huge number of examples on this. Um, a couple that I really like. So, uh, when a rider had a, uh, a rest and recovery day, now usually what the athletes would do, it just meant that they weren't training. So they would just go about their normal day-to-day business. They would play with their kids, they would go and go to the shops, whatever, what he would change. He said, if you're on a rest and recovery day, you sit down or you lie down and you do nothing.

00:15:34

And so that change actually improved their recovery. So the next time they went training, they were actually a little bit better. It even went into other things like when they were staying at hotels, they would take, first of all their own pillows, then their own mattresses and pillows that that actually optimized the rest that they would get the night before a race so that they could, uh, just again, improve their performance, um, the next day. So all of those marginal gains from 2002 to the point of 2008. So the Beijing Olympics, the British cycling team won seven out of 10 gold medals. So a massive transformation. And from there, they've gone on to become the most successful cycling team in the world. So how could we take marginal gains and apply it to what we did? Um, so one of them was improvement themes. And again, you know, there's, there's been documentation on this, but it's a great example of marginal gains.

00:16:30

So we would take things that we wanted to improve and we would sit down and the, the team would then brainstorm, what would that definition of awesome look like if you would be really good at this one thing. Um, then from there we would also as a team brainstorm what were all of the prob problems that we were currently facing. Um, and then from there we would then say, right, we're gonna take some of these. Let's take a, a small set. Um, so breaking it down into very small chunks of the things within our control that we could solve and, and move forward to making it better. And by looping that process again and again, and ticking these things off, then it's amazing kind of looking back at how many of those things you did. So it kind of takes away that overwhelming, uh, that sense of overwhelm, um, that you have by just focusing on small steps, which over time aggregate together make things much better.

00:17:27

Our fourth challenge was gaining buy-in, not only from the team, but also from the business. So, you know, how can you do this? So we, we use the power of stories. So for, um, you may have heard of the anecdote to change a culture that you need to change the stories within that. So to do that, you know, you need to, um, have an, have an inspire. You know, you, you, you, you need to be able to inspire people through having the story to help drive the change. And you need to be able to have a purpose that people can buy into and relate to, to help drive the culture change that is also in line with the organization's values. So to do this, we came up with a number of catchphrases and mantras to help reinforce the messages that we could use again and again.

00:18:14

So here, here are some examples. So for example, never see the same issue twice. We all accept that when developing and deploying software, you're going to see issues. We, we know that happens. But the problem that the, the main thing is to make sure that that problem, that exact problem, does not reoccur again. And to do that, fix a problem, build the right automated tests around it to make sure that that particular piece of functionality can't break the same way again in future, as we're working within a cha trading environment, we came up with the mantra of changing trading from an art to a science. So the more, the better we can monitor and measure our systems, the more systematic we can become. And therefore, as our trading, our trading, our trading itself can become much more systematic as well. So much so that our traders have actually gone on to reuse this quote when speaking in the press.

00:19:06

And you really know that your mantras have taken good effect when they used right back at you. And you can actually, you know, when you're reading them and hearing them used it back at you co complete from, um, to production in an hour. So this is the benchmark we set ourselves so that we could release quickly, either when fixing an issue or when reacting, um, to, you know, a business change that had to be done quickly. We know that we can address the functionality and then roll out that change within a, a quick period of time or from quarterly to daily releases. That really is the aggregation of all the marginal gains that we've gone through to be able to release on a regular frequency and to finish up. It's no fun if it's too easy. So we all know change is hard. We've, you know, probably all the people in this room have been on, you know, big change transformation journeys. So they're not easy. You've gotta persevere with them. They should be fun, they should be challenging. But the more you persevere, then you're going to see success at the end of it.

00:20:08

So let's look at the, the results over that five year period. So we've gone from having 13 systems and managed to tackle a lot of that technical debt to get to three. And so we've, in doing that, we've modernized those platforms. So they're no longer the monoliths that we had there. Modern, uh, microservice, architected, uh, platforms. And we've managed to change from having that kind of quarterly release process to releasing more often than once, once a day. Um, but in doing that, and again, going, uh, to some of the other mantras, um, faster, safer, better, um, we've reduced the number of instances, even though we're releasing more often, the number of incidents has gone down quite substantially. So we've only had one critical incident this year. Um, and so, you know, that is something that is really meaningful for us as a business. 'cause when we have incidents that can mean a reputational damage or fi or major financial loss.

00:21:05

And a lot of that has been achieved through, you know, enhancing our code coverage. So again, these are metrics that we track. Um, so on our whole estate, including, you know, what legacy that remains, it's 60%. But then we also monitor any new code coverage, um, which is much higher. And then our metrics, they go from code complete to production. So it was a week that's now under an hour and is getting faster all the time. And the most profound change is really in the team. So while it was a, a, you know, two separate teams between Devon Ops who are now one team that's combined, um, we have had more investment because we have been, you know, giving more value to the business. So they've been happy to invest more in in us. And so the team has gone up slightly, but the biggest impact is the number of people that are actually eng engaging at a code level. So while it was only 18 before, um, so that was half of our development team were actually writing code of the combined development and support teams. Um, we now have 80% of the people are actually engaging at a code level. So what that then means is that the business are seeing the value of 40 developers rather than, uh, 18 before, um, even if, even though it's kind of been a marginal increase in overall headcount.

00:22:25

So looking at the business results, so you know, we have a successful business. Our, you know, trading volumes have been increasing over time over the five years. It's gone from 600, uh, billion a year to over a trillion. But what could have happened was that in doing that we would've had to increase traders that that would've happened if we'd not changed the systems. But by changing the systems, the trading headcount has actually gone down. So it's gone down by about 25%, and each trader is now capable of handling almost three times the volume, um, than they were able to do before. And that's because we've been able to streamline and automate the processes that they follow. So again, looking at the whole, uh, value proposition, um, across not just tech, but the, the business as well. And then there's no point in doing that if it makes trading performance worse, that is something that we measure very accurately, and that has improved by 24% over, over those, those years since we started.

00:23:23

So just before we started, it was going down, um, since then it's been continually going up. Um, and so that is a, that that benefit goes directly back to our clients. And another benefit that goes to our clients is the trading fees. So as we have got more sophisticated, we need, we then use simpler services from our brokers. They then charge us less for those simple services. And so that has amassed a whopping 47% saving in brokerage fees that we charge. Uh, we get charged and aggregating the financial value of that, uh, together, that is $1 billion in savings for our clients over those years. So it's really substantial, the impact, uh, that, the benefit that, that we've been able to deliver. So then there's actually the, the changing of the business. So Christian West, who's our, our business sponsor, uh, and runs the equities trading desk. So he now thinks of technology differently. So he was quoted in the press saying that the trading team has become a mix of traders, quants and technologists, um, with the latter two becoming a bigger part of the mix. So that is a massive transformation that the business have felt. Um, and so they're operating in a very different mindset than they were before. And in doing all of that, we've been awarded the Best Buy-side trading desk for the last three years ago. So something that we are collectively as a team very proud of.

00:24:47

And just to wrap up with a few key takeaways. So firstly, know your sphere of influence. Stay away from what you can't influence or from what you or from what you can't control. Use of nudge theory. Remember the, remember the, the fly on the, the, the sticker of the fly that can help drive behavioral mindset change and also nudges can, when, when applied can have a huge impact and be very cheap to implement use of marginal gains and improvement themes just by listing out and, and addressing each issue one by one, looking back over time, it can have quite a profound effect on, on the overall amount of deliverables that you're working on. And remember the power of stories and mantras and catchphrases. You know, sometimes reality is too complex and you need to break it down into a story to be able to get people to understand what it is that you're trying to do.

00:25:39

And just a couple of problems that still remain for us. So, you know, in JP Morgan, we're a huge organization, 50,000 technologists. Um, we've got many different technology teams moving at different paces as part of this DevOps transformation that, you know, we are working on across the bank. And we're trying to find ways to make it easy for those teams to adopt through socializing those, um, experiences. And, you know, we're, we're, we're trying to look at how we can socialize experiences in an easy way so that people can, from all parts of the organization, can, can understand what it is to go on that transformation. And, you know, we are, we are always hiring, so we're trying to nurture and hire the right kind of talent so that we can build the teams of tomorrow as part of this journey that we're going on. Thank you very much for listening.