From months to minutes: Modernizing Database Technology at JP Morgan Chase
From months to minutes: Modernizing Database Technology at JP Morgan Chase
Shaun Norris
Director - Cloud Platform Services - Databases, JPMorgan Chase & Co.
Chapters
Full transcript
The complete talk — auto-generated from the talk's captions.
All right, so next up is Sean Norris, who's currently executive director and head of Public Cloud database products at JP Morgan Chase. According to Forbes Magazine, JP Morgan Chase is the largest bank in the US with over $3 trillion in assets under management. They operate at mind boggling scale. They generate over $200 billion in annual revenue with nearly 300,000 employees and over 40,000 developers.
So I first met Sean when he spoke at the DevOps Enterprise Summit in London in 2016, back when he was at Chartered Bank, and I've learned something from every interaction that I've had with him last year. I learned what he was working on and I was so excited about, I was excited for him. And one reason was the enormity of the undertaking that, uh, he's on right now, which is helping to build the platforms that help developers move their database workloads to the cloud. And the other reason is a surprising areas that he's been focusing on, which I suspect many technology organizations have accidentally overlooked.
Here's Sean. He keeps his crew on track for all his legendary adventures. They should put up a plaque for Sean, the sheep. He's shown the sheep, he even looks a doubt with those who killed.
Good morning everyone. I'm Sean Norris, and, uh, thank you Jean for the nice introduction. I lead, uh, public cloud database products for JP Morgan Chase And to introduce myself quickly. I've been a veteran of the IT industry for the last 25 odd years, and, uh, started my career in Canada, but quickly moved to the UK earlier in my career.
And since then I've lived in Switzerland, Argentina, South Africa, Singapore, and now I'm back in the uk. So I think I was a digital nomad before it was a thing. Um, JP Morgan Chase, though of the companies I've worked for, and, you know, those companies include places like a w s and Pivotal, um, startups like last minute.com. But JPMorgan Chase is actually the only firm that I've ever returned to.
So I'm at my second stint at JPMorgan Chase, and this is my sixth DevOps Enterprise Summit in some capacity. And the fourth time speaking. And you know, you're gonna see a nautical theme through this talk. Um, I have imposter syndrome whenever I come into this room, so if you're up after me, you're talking, don't worry, I I have the same, but I've actually realized from doing a bit of boat racing that if you're actually at the back of the fleet, it's not imposter syndrome.
You're just last. So, Through through my career, I started out in ops. So I really identify with the, you know, system administrator of old, I migrated into software engineering roles, and now I really identify as a technology leader, helping teams build great products. I live in Livington in the UK on the south coast, uh, with my wife Mags and, uh, two crazy dogs, Odie and Helix.
But enough about me. Let's talk about this boat. Um, the, the boat you're looking at on the screen here is a boat named Madcap. Now, this is not just any boat, this is what's known as an X one design, and it's only found in the Oland, which is that famous body of water between the Isle of White and, and mainland England.
And if you're familiar, that's where, uh, Nelson's fleet was based. It's where the first America's Cup race was held. It wasn't called the America's Cup then until America won it. And I'm lucky enough to own this boat in that body of water.
But this is not just any X one design boat. This is actually the oldest one still sailing. It was actually built in 1911, uh, out of wood, and it's been restored a couple times in its lifetime. But I thought as putting this talk together, this is a great analogy for heritage technology and heritage is great for the weekends, but really when it comes to day-to-day business operation, we often need something more modern.
So this really is a talk about database. It's not about boats, but, uh, I thought the boats made a better background and better photos. And so we're gonna have some, uh, nautical pictures accompanying this talk about databases. Uh, a couple of reminders.
First, uh, while I work for JP Morgan Chase, my opinions here today are my own and don't necessarily reflect my employer. And if I mention any technologies, please don't take that as an endorsement. Um, it, it is not. Let's talk about the scale of JP Morgan Chase.
Gene introduced it to some extent, but JP Morgan has over 290,000 employees and over 43,000 developers in 2023. Uh, technology spend is estimated to be in the neighborhood of $15.3 billion US we to run that size of a state run, tens of thousands of database instances, thousands of application workloads are in the public cloud. And we, as you might imagine, have a large estate of, uh, relational NoSQL databases. We've made a heavy investment into DevOps, uh, practices as a firm.
And I think maybe one of the best ways to illustrate that is chase.com, which you're likely all familiar with as one of the largest, uh, online banking platforms in the world. We now release code to that on average, 15 times per week. And just a few years ago, that would've been, uh, unusual. Now, databases are a growth market.
You have one, and suddenly you have, you have them all over the place. Now, when we think about databases at JP Morgan Chase, the demand for database tech is high, and it's growing on top of a already high base. And unli, unlike this particular day a few weeks ago where we had no wind when it comes to databases as a firm, the wind is really in our sails. And production databases across the firm in the last 12 months have grown on an instant basis by over 28%.
Now, that's great. It's great to hear me talk about, uh, you know, the great technology we're building. But this is from one of our customers. And so this is from, you know, our customers are internal.
We provide technology platforms to application teams, building applications that all of you and the rest of our customers use. And so this is from Harsh, who's the managing director. Uh, he's the global head of engineering for our digital document services. And they did a migration and they migrated.
They migrated to a new modern database platform. They migrated millions of documents as part of this effort. And this was a joint effort between the application team and our database platform team. And the outcome of that was when they were finished working with us to do this migration, their search performance was 85% better, and transaction update time was 50% better.
And so, uh, you know, harsh here mentions how key the partnership was to getting this done. This was not just a lift and shift onto a new database. And also it was not purely an application effort. It was a really good example of working together to, uh, get to a great outcome.
Now, when we think of where we run our databases, it's really a combination of all three of these areas. Uh, you know, if you think of everything below the line as heritage, um, those would be, uh, traditionally built databases, your artisan handcrafted database as it were. Whereas everything above the line, we think of as modern. And we're gonna get into and talk about, you know, modern versus heritage a little bit.
And you can see a picture of madcap on the left and a, you know, a sort of current day, the state of the art technology of sailing on the right, and they couldn't be more different. And it's actually similar, and I think this is a good analogy for thinking about how we deliver databases, uh, to the firm and to the firm's application builders. Um, you know, it wasn't always the way, uh, in the testimonial we heard from, from harsh, uh, in the past, if you wanted a new database, it could take months. Well, why?
Well, there were a lot of tickets involved, and a lot of different teams had to work together and answer different tickets. And if you didn't fill it in correctly, it might get sent back to you. There were lots of handoffs. They were bespoke, everyone was typically manually configured.
There would be custom patch versions, custom engine version levels. Uh, you'd, you'd need a separate database administration team, a whole team of database experts to run these databases for you. And then even once you got the database, often months later, you needed a separate team to actually do your kind of ongoing operations of that database. And so, you know, that level of effort incidentally, also kind of applies to owning a classic boat.
There's a very high ratio of working on the boat to actually sailing and enjoying the boat. Now, when we think about modern database tech as we provision it today, it's very different. When a application team orders a database, typically it's available in minutes. That's done self-service through an a p i.
It will typically have a standard config, it'll have consistent patch and engine versions across the fleet, and it will be self-managed by the application team to a large degree. And the ongoing operations involved with that database are typically done, uh, through self-service, and they're integrated into our C I C D pipeline tools. So why does database modernization matter? Well, you know, this is, uh, I think another serious outcome.
This is why it's really not optional for us anymore to get to modern technology platforms and modern database platforms. You know, installing a database engine is only about 10% of the job of provisioning and getting a database suit suitable for use within the firm. Hundreds of controls apply in a highly regulated firm such as ours. These controls used to be largely manual, and now they're increasingly automated.
But the, uh, the metrics we have coming back in this case here, that for teams migrating from one of our most commonly used databases, if they went from our heritage platform to our modern cloud platform, same database engine, mind you. So you're not changing the type of database you're running, you're just changing from the heritage platform to the modern one. You were 82% less likely to have a database related major incident. This has been a huge at scale success.
This is over thousands of instances. And so one of the ways that we're looking at it when it comes to, uh, helping application teams manage their own technology is really looking at a split between, uh, container and content. And so, you know, just as a, a boat without a crew is not really much use. Well, a database engine without some data in it or without some content is also not much use.
And so, um, this has been one of the helpful mental models to think about. How do you provision stable ready, fit for purpose technology at scale in a firm as big as ours, and yet make it suitable for use and fit for purpose? So this container content split is one of the ways we think about that. So when you look at container, well, that's, I have the engine installed, I have the right infrastructure, I have C P U disc and network, I have, uh, you know, my operating system and engine patches.
I have replication and resilience and failover setup. And hopefully those are all done through automation with, uh, a p I driven control planes. And then when it comes to content, well, that's my schema and the data in that schema. Uh, how, how are users, uh, authenticating and authorizing, uh, are those apps authorizing?
Are they human users, ss, r e, et cetera? Um, are you publishing your schema to a catalog? Are you enforcing all the relevant controls? But we then typically like to provide standard administration, uh, capabilities through curated self-service, uh, rather than just giving people the ability to log on and do whatever they like.
So it's, it's worth now a look at controls, and you might find this a bit of an odd photo, but I chose this one not because of the massive ropes and the, you know, the cam clutches and, and the cleats here, but if you look carefully, like on the red rope to the right, there's a little yellow piece of twine. And what we do when racing is that's actually a baseline marker. So once we figure out how to make the boat go fast, and, you know, full confession, I haven't quite figured out how to do that yet. But, um, when we do, you wanna mark the rope so you can get back to those settings.
So I thought this is a, a great visual analogy of what we try and do, particularly with controls. Uh, we want to try and make sure that the systems are well-built and in control. So some of the controls we have to look after over and above just the basic database are things like backup reporting, are backups happening? Are they on the right schedule?
Uh, have you tested them? Uh, privileged access management, who logged into a database with what privileges? If those were elevated, what did they do? And was it appropriate?
Uh, security drift monitoring, was the database set up with the right security baselines? And has it drifted over time away from that? And if so, who's putting it back? Um, things like sock one p c i compliance, uh, highly confidential data certification, resilience testing.
You know, the list that goes on and running databases at scale is not without its challenges. You know, one of the challenges of, uh, sailing an old wooden boat is it has no flotation in it. So my friend Simon here on the left, uh, the angle he's at, if he was another 10 degrees over, chances are he would've sunk 'cause he would've taken on too much water into the open cockpit. So this is a little scarier than it looks.
Uh, this was earlier this, uh, year at coss week. Um, but when it comes to running databases, particularly when you put public cloud into the mix, well, cloud is of great benefit, but it's also not without its challenges. Um, how do you get, get the right balance for app teams between re just refactoring or replatforming? To what extent should I have to think about reworking how my app works when I want to go to the cloud?
Because in a lot of cases, or in some cases, the technology that teams are comfortable and have been using for years on premises is actually not available in the public cloud. Um, migrations as well are a big challenge. How do you migrate from when you have 10 or more different source databases and a small set of target databases? How do you make sure anyone can migrate from any to any safely, you know, repeatedly with rollback, et cetera.
And So public cloud migration has been not without its challenges, but we are, are moving quickly in that direction. So let's talk a little bit about then, we've talked about migration. Now let's talk about new applications. So we also wanna take a longer horizon view.
If, if you're coming as a new application builder, well, what sort of things should you be considering when you choose what is the right data store for your application? So we put together this, and we've kind of put it in the form of the Agile manifesto, and I won't read the whole thing, but to go through this. We, we, we don't want this to be binary that you should only ever use open source and never use commercial databases, for example, you know, when it comes to open source, we are bullish on open source. We think it's really important as a firm, we already run thousands of open source database instances.
Um, but on the balance side, commercial databases are still important to us. We will still run and support large numbers of those into the future. And then convenience versus portability, that's, uh, another area. Cross-cloud.
Uh, some folks like to chuckle about multi-cloud in our, uh, industry. It's a reality. Cross-cloud, in my opinion, is going to grow in importance. And so if you have coded your application to be tightly embedded and will only work in one cloud, if you for whatever reason need to migrate away, um, how are you going to do that?
Uh, what level of difficulty and what timescale is that going to be on? One of the mitigations that we are looking at, and we're quite bullish on, is the idea of using Postgres compatibility as a mitigation and other considerations. If you, uh, code heavily against the identity and access management substrate of one cloud, and then you need to move to another, um, how much work are you going to need to redo? If all of your developers are skilled up and only know the, uh, specific skills that work on one cloud, how portable is that going to be if you decide to onboard to another?
These are all open questions still. So let's move on and, and wrap things up and look at, well, what's worked so far in, in our environment? Well, first of all, a contribution model. When we're building a cloud platform and the databases on that cloud platform, often there are specific, uh, capabilities that a, a certain team will want but are not coming soon enough to meet their migration needs.
And we're rapidly embracing a model that allows those teams to come work with us and contribute features more quickly than they would be available otherwise. We, uh, have successfully invested in a firm-wide C I C D tool chain. So whatever line of business you're in, whatever development team you work in, it's a consistent set of tools so that you can build tasks, deploy your code with confidence, um, visible work. I want to give a shout out to Dominica de Grandis, who I met briefly earlier.
Um, we've successfully made a lot of our work visible. We have one central, uh, clearing house for, uh, raising a cloud feature request. If you need something to migrate your application, we've successfully got that into a one-stop shop where you can raise that and ask for it. So, um, all those things have worked for us so far.
If we talk about, well, where we started, well, Python, we had a lot of database administrators looking who were keen to retrain as software engineers. So we've used Python as a training vehicle internally, uh, with a lot of success. We have trained hundreds of DBAs and retrain them as, as cloud and software engineers. And, uh, cloud skills training, we've doubled down on, we've focused on both open source certifications like c a d as well as cloud specific, uh, public cloud certifications.
And we've done that at scale with, you know, thousands of certifications across the firm. 'em, and if we look at, you know, help, we're looking for, well, there's a need for more community and industry standards. I mentioned things like security configuration, baselines. Well, those are bespoke and firms specific today.
I, I believe there's a need to come together as an industry and actually open source some of these ideas and actually collaborate for greater, uh, you know, uh, cross collaboration across the industry. And there's a need to shift compliance further left. You know, we, we've started and we need to go further and we need to get compliance built into our products and our applications right up front as part of it. And, and, and we need to go more in that direction.
And then the last thing we need is really feedback. If you've enjoyed this, if it's sparked any conversation ideas, uh, if you want to chat more, you can engage with me on LinkedIn, you can, you can, uh, chat with me at the conference. And lastly, uh, as we wrap this up, it's time to drop our sails on this talk and to, uh, take the boat and the talk back to the mooring. But before I go, a few word of thanks, first of all, to Gene and the organizers.
Thanks for organizing an amazing conference. Um, to Emily and the rest of the media relations team at JPMorgan Chase, it's a huge effort to be able to come and share at a public event like this. So they did a fantastic job in, in helping me get here and share this with you, uh, to my wife Mags, for taking most of the photos and most of all to all of you for making this an amazing community where I learn a ton every year. Thank you very much.
He.