Shaping the Cloud - How We Transformed FINRA With DevOps

My name is Daniel Koo. I'm a senior director at FINRA, currently managing the DevOps products and engineering organization. I strive to be a visionary leader, influencer, and a change agent wherever I am and wherever I belong to.

So I led the DevOps movement at FINRA since 2015, and I'm always looking for ways to positively disrupt the enterprise and then contribute back to the industry.

All right, so brief history about myself, where I started and where I am now. So prior to 2007, I was a software developer. I worked for various different companies, pretty much coding any task that was given to me.

And then in 2007, I joined FINRA as a contractor, and I joined FINRA to help with the modern test automation development using Selenium and Java.

So I played that role for the next two to three years, and then FINRA appointed me as the automation architect. They wanted to see what I have built for several projects and now scale across the enterprise.

And in 2012 is when I joined FINRA as a full-time employee, and then in 2015 was when I was asked to lead a group called Productivity Engineering. That's when we start to think about DevOps and CI/CD.

And then in 2017, my team expanded. We grew, and then we were tasked to implement DevOps across the enterprise, and here I am now.

So quick intro on FINRA. So who has heard about FINRA here? Oh, quite a few. Wow. So FINRA stands for Financial Industry Regulatory Authority. We oversee the broker-dealer industry in the U.S., and we regulate the markets. So we have about 4,000 brokerage firms that we regulate, more than 634,000 securities representatives, and we monitor 12 different markets and exchanges. Our mission is to protect investors and promote market integrity. We are a not-for-profit organization. We have close to $1 billion per year in revenue, and we have total of 3,600 employees, and in tech organization, we have around 1,200 people.

So FINRA is a big data company. We ingest up to 135 billion transactions in a single day. When we reconstruct these data, we can get up to trillions of edges and nodes, and our storage footprint is more than 30 petabytes of data. And we have more than 150 different applications that we deploy in AWS to run our regulatory function. And we could be running up to 50,000 compute nodes per day. Not per month, not per year, but per day.

So here is a quick snapshot of our data volume growth. In 2017, we had about 37 billion events on average per day, and now it's grown up to 135 billion transactions in a single day. So this shows how much data that we're ingesting, how much we're analyzing, and how much data we're storing to do our regulatory function.

All right, so let's talk DevOps now. Our journey really began back in 2014 when we started to think about cloud. So we decided to move to the cloud with the growing demand of the data volume. So we couldn't sustain the data anymore in the data center on-prem. So that's when we decided to move to the cloud, and when we think about moving to the cloud, we had to change our mindset. We couldn't continue to practice the way we were doing in the data center, right? Having the ops team, the infrastructure team, provision the servers, and then we deploy the applications onto the servers, right? We had to change our mindset. How can we, in the cloud, provision, configure, deploy, and then automate, operate? So we had to rethink, right? We had to reestablish our CI/CD strategy as well.

So that's when, in 2015, a group was formed called Productivity Engineering. It was a small team that was formed to think about what are the gaps. So we were already practicing CI to a good level, but then when we think about, again, cloud and infrastructure automation, we had to think about what gaps do we have within our current tool chain, right? We're using things like Jenkins, back then SVN, Git, right? We're doing that continuous integration, but again, we thought there were gaps in terms of our tool chain. So a lot of different entities got together, including architects, security, operations, and development community got together to think about some of the products that we need to build to practice DevOps.

So in 2015 and 2016 is when we really cranked up building these products, right? These products were onboarding applications, provisioning, setting up your network, your security groups, how to do deployment in the cloud. And when we were building these products, we had to think about baking in security, baking in compliance, and then following architectural patterns. We didn't just go and create any products, right? We had to think about us being a regulatory company. We really need to think about compliance and security, so we had to build all those things into the tools.

Now, we built these tools, and we saw some projects starting to adopt, right? So we start to see some success, but then it wasn't scaling, right? We have 150 different applications that are trying to move to the cloud, right, migrate to the cloud, but it just wasn't scaling enough. So what we decided to do in 2017 was to transform our software configuration management group, the CMs, into application engineering group. These guys became the experts in DevOps, experts in these tools and the products that we custom developed and we maintain, and they were embedded within the project teams. Now, it wasn't that these guys were doing all the DevOps work. That was not our point. That was not our goal. It was for these guys to go and help teams to practice DevOps. That was the differentiator.

Now in 2018, we really wanted to scale across the enterprise, right? We were really serious about moving all of our systems to the cloud. That's when we invested a lot in training. So we built a self-paced boot camp training website for developers to come anytime at their own pace and take the boot camp training, where we teach about the concepts and the tools and how to practice DevOps. So by 2018, we had about 600 developers that took the course, and they were able to actually start practicing DevOps.

And then 2018 was the year when we started to move a good portion of our portfolio to containers. We started to move to Docker, started to move to AWS ECS. So a good portion of our portfolio got migrated over to containers. And in 2019, this year, we start to look at serverless. Industry was moving to serverless. And we were seeing a high maintenance cost even moving to the containers, right? So we start to think about serverless using Lambdas, Fargate, and Aurora. So we start to see a small movement of apps to serverless.

And also this year, we really start to look at AI/ML. So within our business application portfolio, there were apps that were already adopting AI/ML and seeing value. So what we wanted to do was to look at what can we do within the DevOps space. How can we apply AI/ML to DevOps? So that's where we are now.

So I want to take a moment to read our vision statement. I truly believe in having a vision statement and mission. It really helps the organization to understand where we're trying to go and what we need to do.

So we have four vision statements within my organization. First is to enable teams to deliver software faster and ensure reliability at enterprise scale through automation, building products with built-in security, compliance best practices, and continuous monitoring.

The second one is advocate DevOps practices that allow teams to gain confidence in delivery pipeline and empower them to continuously deploy to production on their own anytime, anywhere.

Third, provide feedback mechanism to teams by continuously collecting data and making them accessible via meaningful interfaces.

And lastly, promote collaboration and innovation by shaping the DevOps culture within the technology community. So now are we meeting all these goals? Not yet, but this really helps us to drive and steer the way we want to go.

So let's talk about the organizational structure. I touched on some of the groups already, but the DevOps engineer practice at FINRA consists of these three different groups. First is the productivity engineering group.

So my entire organization is about 50 people serving 1,200 technologists. So we have about 20% of the folks working within the productivity engineering, focusing on building custom tools and maintaining developer tools like Jenkins, Bitbucket, Jira, Confluence, and et cetera. So these guys are full stack developers. They can develop front-end, back-ends. They are DevOps and test capable.

The second group is called cloud engineering. So they are primarily responsible for releasing the base images, the Docker images, and the AMIs, Amazon Machine Images, that bakes in all the compliance, all the security, and all the governance tools that are needed to run our applications. And they are system engineers with programming knowledge, and they are 20% of the organization.

And lastly, the application engineering group. So I briefly touched upon those guys earlier. These guys are the field engineers that are on the floor helping the business application teams. They are the experts in CI/CD, DevOps, and SRE, and they are the automation guru. 60% of my organization are application engineers, and without this group, it was not possible for us to scale DevOps across the enterprise.

So let's talk about toolchain. I'm going to highlight some of our custom tools. There are a lot of tools that we use, but I'm going to highlight some of the custom tools that we built that really helped us to implement DevOps.

And I want to speak in four different phases. These are the phases that project teams typically go through. Inception phase is when teams start thinking about building their app, or they want to migrate their app to the cloud. Kickstart is when they start coding, and then development testing is a cycle that they go through before production. And finally released is after they go to production is our monitoring and governance.

So let's talk about the inception phase. We built a tool called Onboard. This tool basically sets up all of your access to AWS. It creates your active directory group, your key pairs, your IAM roles, your certificates, your token to make dynamic DNS registration, your Jira project, your Confluence space, your Bitbucket project, your Jenkins setup. So everything is done through this application. It used to take us a month, sometimes even two months between different groups to get all of this set up. Now it happens in one day.

Second app is called Portus. This is an app that we created in conjunction with our information security. This is an app that helps teams to manage their security groups in AWS. So InfoSec would come and create the policies, right? They would whitelist the rules. And then now the development team can come into this app self-service and start creating their security groups using infrastructure as code.

Now, the kickstart phase is when they start coding. And when I talk coding, this is both infrastructure code and application code. That's the difference, right? When we move to the cloud and think about DevOps. So we spend a lot of time thinking about how we can create resources in AWS in a compliant, secure manner, as well as putting in the architecture patterns that we want to see in our infrastructure. So we created a tool called Provision. Underneath it uses CloudFormation and various APIs and SDKs to create the resources in AWS. But again, we define different stacks, right? We define the patterns. For a traditional web application, you use this type, it will go ahead and create all your resources for you and configure it for you. That was the goal for Provision.

AppConfig was the Puppet and Ansible module that we created to configure your servers.

F3, that is a product that we built on top of Jenkins. I'm sure everyone knows about Jenkins here. So what we created was to help teams to build their build jobs, their deployment job, their pipeline, and orchestration, again, in a compliant, secure, convenient manner. And Docker base images and base AMIs is what we produce so that it's got everything that's needed for you to run your application built in.

Now you get into the development testing cycle, right? So you use the same tools to continue to build your application. And then there are some other tools that we built. We have a tool called Fidelius, which manages your secrets in AWS. Now it's ready, it's in production, and then we have different tools that we use to monitor and govern our infrastructure, right? Both open source, third party, and our custom tools to achieve that.

These are the goals that we try to integrate into our tools, whatever we build, whatever we choose. Standardization, very important. Make it convenient, make it easy for people. Compliance, building security in from the beginning, not at the end, but from the beginning. Integrating Ops from the beginning, embedding architectural patterns and self-service and automation all the way through.

So we have many open source projects out on GitHub. So I encourage everyone to go check it out. We also have other tools that are coming out as well.

All right. So we talked about our history, our tool chain, our organizational structure. Now I want to touch upon some of the best practices. I'm sure many of you guys are already aware of this, but I want to touch on these six items. First, automate everything. This was our motto going into DevOps and CI/CD, is to try to automate everything that we can. I know we can't automate everything, but having this mindset helps us to think about when we're dealing with a problem or when we try to solve a problem, right, we always think about automating that. Infrastructure as code, configuration as code. We have a lot of APIs that we can integrate, right? Automated tests and rollback.

Second thing, compliance and security. Many times you think about compliance and security at the end, but you need to be thinking about compliance security from the beginning. And also a key thing here is baking in the compliance and security in the tools itself, right? No opting out. So when teams use a tool, you already have compliance and security built in, encryption at rest and in transit. Authentication, authorization on every traffic and making sure you have scans in place. Very important.

Third thing is standardization and architectural patterns. Right? I talked about that when we described the tools. We define different stacks, we define different types, and then let them use those types, right? Make it easy for people. Make it convenient for people, right? Easy to troubleshoot. No reinventing the wheel. That's what we try to do.

Fourth, resiliency and reliability. Moving to the cloud, again, we have to change our mindset. We have to design for resiliency, right? Cloud, your AZ could go down, your service code could go down, your service could just go down. So you have to design for resiliency and think about auto-recovery.

And then the delivery insights. So we try to capture all the relevant data within the pipeline and use that to analyze how we're doing predictions, what we want to do next. So you've got to make sure you define the different data that you want to collect and catalog them, and use them to query, analyze, and visualize.

And lastly, monitoring governance. Similar to compliance and security, think about monitoring governance from the beginning, not at the end. Pre-configure them, bake them into the tools. We want to have transparency. We want to be audit friendly and audit ready. So these are the different best practices that we try to follow.

So I want to show you how we're practicing this within FINRA, with a simple workflow.

So first, we have different entities working together to define the policies and standards. We have security, enterprise architecture. We could have development community creating the policies and standards.

And then those get fed into the product team that I talked about. The productivity engineering team is the one that takes these standards and policies and creates the products.

Once they're created, with the help of application engineers, development teams consume these tools, and then they start deploying to the different lifecycle accounts that we have.

And after that, because we have monitoring and governance in place, we can see what is happening within our infrastructure. What are the changes that are happening? And even at some occasions, we roll back to the configurations that we have set.

And now with that, we can create compliance reports that can be used by audits, and it gets fed back into, again, the security and architecture and development community. We refine the policies and standards. Now, that goes into the products. We make enhancements and the necessary changes, and then you go into the cycle. And that is how we're practicing those best practices that I defined.

All right, so now we've got different tools and the products that we developed, and teams are starting to use it. But how do we scale? How do we see this across the board?

Again, our goal was to enable teams. My organization's goal was to enable teams to practice DevOps. We don't have a separate DevOps team that is doing all the DevOps work, but the goal is to make developers practice DevOps, and we also provide oversight. We need to become the feedback channel to listen to the developers. What are they going through? What problems? What do they want to see? So you have the focused product team developing the products, the embedded engineers helping the developers and getting the feedback.

And also continuously training these folks so that they are DevOps aware.

So speaking of training, this is the self-paced boot camp that I mentioned before. So we created on the website that the developers can come and this is a purely lab-based exercises. So it's not just theories and concepts. But they can actually come and do lab exercises.

The goal was, at the end of this training, that the developers can now go and practice DevOps. Not just with knowledge, but they have hands-on experience using our tools and understanding our concepts. So we teach things like tools and the best practices in containers, serverless, database deployments, security, and operations. So we cover all the fundamentals of the DevOps practice through this training.

Okay, so now we're seeing DevOps being scaled across the enterprise. Now, how do we measure the maturity? We don't want to just stay stagnant, stay at the same level. We want to continuously measure how we're maturing. So six different aspects of measurement here. The automation level, we want to make sure teams are using approved CI/CD solutions. No manual steps whatsoever. No downtime. Do they have adequate test coverage? Do they have health check in place? So these are the things that we try to measure for each of the projects.

Tool adoption, are they using approved solutions? If so, what version are they using? Because as we ship out new products, some teams are not upgrading. So we got to make sure as we release new features, new enhancements, that they do adopt the new versions.

CI/CD metrics from beginning to the end, from source code all the way to deployment, we try to measure and collect relevant data. From source code perspective, how many check-ins are they doing? Are they checking in every day? Are they checking in at the end of the sprint? How many builds are they doing? How many failures do they have? How much are they deploying? How much failures do they have? So all these things is what we try to collect, and we analyze to see their maturity.

Next thing is called scorecard. This is what we developed in-house. This is what we measure, the compliance aspect, quality assurance, security, and operations. So what happens is when teams are trying to go to production, we run the scorecard. Right? We run the scorecard and go, "All right, your score is X, right? But you need to reach Y in order for you to deploy into production." Right? So teams are forced to up their score so they can actually deploy to production.

Deployment. We look at their deployment strategies. Are they using things like blue-green? Do they have automated rollback, zero-downtime deployment? Do they have orchestration? Right? All these things is what we look at and see if teams are implementing any of these deployment strategies.

Lastly, reliability and availability. We look at the outage numbers, right? How much downtime do they have in their servers? Are they auto-scaling? Are they resilient? Do they have the right alerts set up? Are they multi-AZ, right? How good is their performance? So all of these elements and aspects is what we look at when we measure the reliability and availability. So it's important for us to look at all of these measurement to see how mature they are, that we continue to get to the next level.

All right, so this is the last slide. What problems still remain? What are we trying to solve now? So we are continuing to shape the future.

One of the things that we're looking at is continuous deployment. I think we're doing pretty good on CI/CD and DevOps, but how can we get to the level we're continuously deploying to production? Right? We're building things like the production deployment readiness check, doing dry runs in production, doing auto upgrades, blue-green, zero-downtime, feature flagging, all the different deployment strategies is what we're trying to implement.

And also using bots, using AI, right? Less humans, use more bots. Apply ChatOps to promote collaboration and being transparent. This is one area that we're looking at now, which we haven't solved yet.

DevOps insight. I talked about being able to collect relevant data to analyze, right? So what we're trying to do is to collect the data to be able to predict different patterns, right? Predict failures before it happens. Applying AI/ML is what we're trying to do with the data that we collect. There is a lot of different things that we can do with AI/ML, and it's not just a buzzword that industry is talking about. I truly believe in using AI/ML in DevOps. I think that's really the next phase within the DevOps in the industry.

And with that, I want to thank everyone for listening. And like I said before, I hope you guys have something to take back to your organization. And please feel free to reach out to me below, my email, or you can find me on LinkedIn. I would love to continue our conversation. Thank you very much.