Controlled Acceleration: How Morgan Stanley Is Evolving SDLC Controls for an Agentic World

Host Intro (Gene Kim)

So one of the hallmarks of great technology leaders is that they liberate developers to do their best work — and all technologists. They free them from burdensome compliance and change approval processes that can make the easy things difficult and the difficult things seem impossible.

And so one of my best examples of this over the years was from Gus Paul, an executive director at Morgan Stanley, founded in the year 1935, $70 billion in revenue, with over $9 trillion in client assets, with over 23,000 technologists. And in 2022, Gus presented how we started off on the trading floor, where developers worked side by side with the traders, and how after the great financial crisis in 2008, the controls piled up until the average change took over three and a half days to approve. And thus began a decades-long journey to create the conditions where developers could do their best work.

They even trained a model which actually analyzed deployment history, and they found that they could predict which changes would cause the least number of customer problems based on the size of the change, how much it automated past incidents, and so forth. And one of my favorite parts was that they ran a six-month pilot where changes approved by the model caused zero incidents, where the changes reviewed by humans had a 1.5% incident rate, and critical fixes went down from two weeks to an hour. And I love this story so much it actually went into the Wiring the Winning Organization book, and it seems so relevant now in an age where coding agents can generate 100 times more code. And the question becomes, how do we safely review that code, ensure that it can go into production securely?

And so up next is one of Gus's colleagues, Kal Elsawaf, an executive director and the global technology lead for the secured financing technology renovation at Morgan Stanley. His group was one of the biggest adopters of the change process that Gus's team pioneered. He will be sharing how his group created the next chapter, showing their amazing work creating the new software development processes that meet this crazy moment in software. Here's Kal.

Khalid Elsawaf

All right. Thanks, everyone. Interesting choice of music.

All right, so my name is Kal. I am an executive director at Morgan Stanley.

My job is really, really interesting. I am responsible for the tech strategy for one of the biggest renovations taking place at Morgan Stanley. We are trying to get off the mainframe. So what Dustin was talking about earlier hits very, very close to home for me.

What I'm going to do today is walk you through how Morgan Stanley is approaching the evolution of our software controls in the age of AI, by walking you through first examples out of my team, and then how that is being extrapolated into themes across the firm.

01Morgan Stanley Tech at Scale

So first, a little bit about Morgan Stanley and tech at the firm. We're about 23,000-plus technologists across the globe. About 21,000 have the right to commit code. So really, really large tech footprint. We do about 15 million annual software builds, and that number has been growing exponentially over the last few years. And we support about 10 billion transactions annually. So we're operating at a really, really massive scale.

And our tech stack is incredibly diverse. We have close-to-hardware FPGAs running at exchanges, all the way up to AI solutions for our wealth management financial advisors. So really the entire gamut of low level all the way to the most advanced use cases you could think of, and pretty much every single programming language that you can think of.

And so our job in tech is really simple. We enable the business. We enable the business to generate revenue. We enable the business to reduce risk. And our job is to get the best out of our technologists.

02The Challenge: Accelerating Safely in a Regulated Environment

So what's the challenge that we're facing today? Well, AI has completely disrupted the game. I don't need to go into depth about that — everyone here knows that, otherwise you'd probably not be here today.

But what's really, really interesting about this transformation when you compare it to DevOps or Agile or cloud is that this one seems to be coming from the business. As much as technologists are really, really excited about the idea of using AI, the business is the one that is really, really driving the adoption of AI. They're the ones really asking us to accelerate. They're the ones asking us to go faster, to deliver new capabilities in less time.

But the thing about Morgan Stanley is we are an incredibly highly regulated environment. We are not the "go fast and break things" type. We are under tremendous amounts of scrutiny from regulators in every jurisdiction that we operate. So we don't want to accidentally end up in the news for the wrong reasons. That's something we actually take a lot of pride in. We are not the types who end up in the news for the wrong reasons.

So the challenge really is: how do we accelerate without introducing undue risk?

And the thing that I'm very, very proud of, working for Morgan Stanley for the last 12 and a half years, is that when we talk about having the best and the brightest from tech, it is really true. We are not dinosaurs. We are on the cutting edge. We've been leveraging AI for as long as we can. And it really has transformed the way we deliver software.

03My Story: Code Reviews as a Bottleneck

So let me tell you a little bit about my story. Two years ago, I was in finance technology. I was moving away from management. I was trying to become an IC again, really focusing on DevOps. DevOps has been my passion for the last 20 years. I've been doing DevOps before I knew what DevOps was. It is my favorite part of the job. For some weird reason, I really, really like having technologists who are incredibly demanding as my clients. I don't know what's wrong with me.

So 2024, I was asked to move into my current team to lead our secured financing technology renovation. And the way the job was presented to me was, pardon my French, "This is a s**t show. You need to go and fix it."

And a big, big part of what I do is code reviews. I am responsible for everything tech within our platform and within our fleet — SDLC, performance, standards, code quality, everything that has to do with tech comes to me. So it's a really, really daunting ask.

A very, very big part of what I do is code reviews. I would say when I took over the role, about 50% of my time went to code reviews. Now, I don't know about you guys — I did not get into this business to only do code reviews. I like to code. So it was a really, really daunting thing.

Now, over the course of 18 months, because of how high priority this program was — this is a program that's being tracked by the board of directors at Morgan Stanley — our team tripled in size. And, well, you can only see one of me. My clones are not off in front of my laptop working. So there's absolutely no way that I could keep up the pace. Reviews were becoming the bottleneck. It was becoming really, really untenable for us to keep track. We know that manual code reviews are very, very subjective. And so we ended up in a situation where something had to be done. How do we maintain a certain level of quality when the output was increasing? And obviously with AI, that just became even more exacerbated.

04Four Themes for Evolving Controls

So what I'm going to talk about is four major themes that are driving our approach to evolving controls. I'm going to walk you through examples from my team, and then show you how that is being spread across the firm.

05Theme One: Automate Your Controls

So what was the first thing that I did when I got access to some of these frontier coding tools? I wanted to make myself more efficient. Code reviews, as I said, were 50% of my time, so I really, really needed to focus on that. So the first thing that we did was we looked at what do we do in a code review — or what do I do in a code review? I was focusing on myself first.

I look for code quality, I look for testing, I look for performance bottlenecks, and I look for architecture drift. My role as a tech lead involves maintaining the strategic architecture. So I need to make sure that with every code change, we are not diverging from our strategic architecture. So in doing so, I came up with a set of skills that breaks down the code review into each of these targeted focus points.

And as we adopted spec-driven development, I also created skills to look at the quality of our specs. And it became a really fun situation where in the code review, we can now tie the spec to the code, to the test, to the architecture — bringing all of that together.

And now I can run a review skill that will look at a code change, figure all this stuff out, and post it. And so what used to be a very manual process now became fully automated. I start my day, I run a script, I extract all the pull requests that need my attention, I feed them into my coding agent, I go get some coffee, talk to some colleagues, and when I come back, the code review is done, the comments are posted, and my developers have something to action.

So how does this apply to the entire firm? Well, we're not just automating the delivery, right? As Gene was talking about with my colleague Gus, really focusing on developer productivity — we have automated the controls as well. Every time we want to make developers more effective, we have to keep in mind that in a highly regulated environment, it is essential that we have controls in place. But our automation and our acceleration is only as efficient — or as slow — as our controls. If controls are interfering with our developers' ability to get a change into production, then we really need to optimize that. So we're really focusing on automating more and more of our controls, getting them into the CI/CD in a way that our developers can really take advantage of.

Takeaway from Theme One: don't just automate the delivery, automate your controls as well. Absolutely essential.

06Theme Two: Shift Controls into the Developer Workflow

So what happened next? After I started running these skills myself, my developers came back to me and said, "Hey, this is really cool. This is really helpful. You're finding edge cases in our code that we are not able to find ourselves. This is really valuable. Can we run your skills ourselves?" I said, "Well, yeah, absolutely." So what I did is I source-controlled the skills so that we can distribute them to our developers, and they can use them on their pull requests themselves. So I'm no longer the bottleneck, and my developers now have the benefit of having me on every single one of their code reviews.

But we actually took it one step further and said, "Well, wait a minute. Why wait until the pull request? Why can't we run the review locally in the workspace — where the developers want to live, right, in their IDEs and their agentic terminals?" So that's what we did next. We said, "Okay, well, this is fantastic. Our developers really do want to deliver high-quality code, so let's enable them to do that." So we transformed the skills to be able to run them where the developers live.

And this is one of the biggest takeaways that I've had from DevOps in general, which is: if you want to scale and you want to be successful in transforming the way a developer operates, you need to bring it to them where they live. It has to be in their IDE, it has to be in their terminal, it has to be in their tools. So we move the control from outside of where a developer wants to operate to where they want to live, inside of their comfort zone. And that is absolutely essential.

Theme Two: shift controls into the developer workflow.

I talked a little bit about how one of the skills is doing architectural drift detection. We're building that on an industry-wide standard that was developed through FINOS, the Finance Open Source organization, called CALM — Common Architecture Language Model. And if you're interested in learning more, come find me afterwards. I am a big advocate of this. If you search for my name and CALM, you'll find a couple of presentations that I've done at FINOS conferences, leveraging architecture as code.

Not in Visios, not in PowerPoints, not in any other out-of-band tool — really having it as code has become essential for us to maintain our strategic architecture with every single code change that we do.

We've integrated DevSecOps as well, so we run security scanning with every change. We can identify if you're using out-of-date or vulnerable libraries. So now as a developer, you get that feedback even before you raise a pull request.

And as Gene was talking about, my colleague Gus Paul came here and talked about a revolutionary concept called systematic change, which completely changed the way we deliver low-risk changes to production. At the core of it is something called the data-driven risk assessment — a standardized, multi-data-point way of assessing risk for every change. And so what I'm really excited about is now we're looking at how to bring that into the pull request, not just on the change that you want to push to production.

07Theme Three: Standardize the Development Process

So we talked about automation, we talked about shifting controls left into developer workflows. Once we did all of that, my developers came back and said, "Well, the review is finding stuff, and it's great, and we're really happy with that, but our code base is scattered all over the place. It's got different standards. It's got different approaches to doing what should be common functionality."

So I took that as a challenge, and as I was on the train going from Montreal to Toronto to speak at a FINOS conference about CALM, I fired off an agent and I basically gave it a really basic instruction: look through our code and come up with a set of standards that can be used as skills for future development.

How many people in this room think that that was a success?

I was actually really impressed at how detailed a return it gave me, telling me, "You have multiple standards in your code base." And it then identified all of the different conflicts. It was really cool because I looked at all the conflicts — and it's a five-hour train ride, so I had time — and I was able to pick out the standard that I felt really was the best. But in doing so, it also helped me identify where we had divergences.

And so by coming up with these skills for all of our future development, our developers know what best practice they should follow. But now we have this incredibly long — I won't tell you how many items because it's depressing — but we have this incredibly large backlog of divergences that are now documented, and that we can tackle as tech debt.

So what we did is really identify those standards that we wanted to be at the base of our development. We turned those into skills. We created a harness agent on top of that. So now every single time that we want to deliver something to production, developers are building off of that harness.

Theme Three: standardize the development process.

One of the really interesting things with AI is — and I'm going to overly simplify this, so I really do apologize to anyone who is deep into model design — AI works really well when you have established patterns. So if you don't have a standard code base, if you don't have a standard developer workflow, it struggles. And this was actually really interesting for us, because what was initially viewed as a "nice to have when we get into support mode" — standard code that we can debug easily — now became a blocker for AI adoption. If we don't standardize our code base, if we don't standardize our process, we will not get the ROI we want from our AI investment. So now it became absolutely essential. What was a nice to have now became imperative.

And again, think about what Dustin was saying earlier — that crisis that sparks some creativity. One of my colleagues used to say, "Never waste a crisis." So when I had the opportunity, and I was pushing really hard for this standardization, and it was always met with "Yeah, but... yeah, but..." — I said, "Okay, that's fine. You're not going to get the best out of AI." And all of a sudden everybody stood up. So it was really, really great to get that out of it. Standardization is absolutely, absolutely essential to what we're doing.

Now, what's really, really interesting for Morgan Stanley is that while we're trying to standardize across the firm, given our place in financial services and our position with that FINOS organization that I talked about, we're trying to standardize across the entire industry. So my colleague Erin Cyril, who could not be here with us today, is co-leading our SDLC common controls framework working group — or common controls catalog — where we're really trying to identify the set of controls that every single bank has to evidence in SDLC, coming up with common nomenclature, common approaches, and common solutions so that each bank doesn't have to reinvent the wheel. So this is a really, really exciting area that Morgan Stanley is pioneering.

08Theme Four: Question Everything

So three themes so far — automation, shifting left, and standardizing development practices. All three of these themes are focused on controls implementation, right? How do we adapt our controls to operate at the scale of AI? But the fourth theme is probably the one I'm most excited about.

The fourth theme: we're questioning everything.

And when I say everything, I really, really do mean everything. We are questioning whether the existing controls make sense. We're questioning whether the risks that they were trying to mitigate still exist. A lot of our controls and our policies and our procedures were written with the assumption that code is written by human beings. That is no longer the case. The vast majority of code today is written either by AI or by at-scale deterministic tools. Which basically means everything that we've been doing for decades is up for reconsideration.

And I absolutely love the Peter Drucker quote: "There is nothing so useless as doing efficiently that which should not be done at all." I love this quote so much because it is the driver behind basically everything that I do. I've been very fortunate at Morgan Stanley to drive a lot of transformations, and every single time someone comes to me with a proposal, I say, "Do we really need to do that? What's the business process? What's the value we're getting out of it?" I also really like this quote because my boss loves sharing it at every single town hall and every single visit. So this is my shout-out to him that I do actually listen.

So it's a really, really fun time, because there's no constraints anymore. The biggest prompt for me when it comes to thinking about our controls is when someone says, "Yeah, but that's what we've always done." But does it have to be how we do things today? So this is a really, really fun part. We're getting the creative juices flowing.

09Recap and What Comes Next

So let's recap. We talked about automation of controls. We talked about shifting left into developer workflows. We talked about standardization — you can't automate what you don't standardize. And we talked about questioning everything.

So what comes next? I always love talking about software delivery as a conveyor belt. I don't know if you have heard of the show How It's Made — it's one of my favorite shows. My daughter and I watch it all the time. We watch it for two very different reasons. She loves watching candy coming out. I'm just amazed at the complexity of the machinery, where it takes these raw inputs, transforms everything, and produces these outputs in this standard, controlled, predictable way. But what really gets my attention is all the checks that happen throughout that conveyor belt — all the little controls, all the little quality tests that exist throughout. It really, really does — I'm passionate about that.

And that's how I see software delivery in the future. You take your change, you put it on the conveyor belt. It looks at everything that matters from a control perspective. It looks at testing, it looks at code quality, it looks at PII data, it looks at security, it looks at everything. And as it's checking all those different steps, it's gathering the evidence automatically. So no more spreadsheets. Dustin, you're absolutely right — spreadsheets are terrible. No spreadsheets, no PDFs, no attestations in other systems. Everything is coming out of your CI/CD. One of my colleagues absolutely loves saying, "Code is king," and I 100% agree with him.

So where are we trying to go? We're trying to go to a standardized approach to calculating risk for PRs, built on the success that we had with systematic changes — sorry. We're reviewing our SDLC policy, and we are introducing continuous assurance. Not every control needs to be a gate. Not every control needs to prevent a change from going to prod. Some of them can happen out of band. So we're investing in these three areas.

10Call to Action

And so I need some help, folks. If you are interested in controls engineering, if you are interested in evolving controls in this AI world, if you're interested in collaborating on how we can come up with a standard for assessing risk on PRs, if you're facing similar challenges in your industry — whether you are a regulated industry or non-regulated — and if you just want to learn more, please come find me. I absolutely love talking about this.

Gene introduced me as the global tech lead for a renovation program. My other big hat is I represent my division on SDLC controls committees, and hence every change to policy and procedure I have to sign off on. A little bit of pressure, but it's something I absolutely, absolutely love. So please come find me. I hope you found this useful, and thank you very much.