Optimizing at Scale: Using ML to Optimize All Applications Across the Service Delivery Platform

Hello, and welcome to our session. My name is Peter Nikolov, and I'm the chief technology officer of Opsani.

In this session, you will learn what continuous optimization as a service is and how to keep your applications at high performance with low cloud spend, and at the same time, have more time to implement value-adding features.

Let me share with you an interesting fact. There are 7.5 quintillion grains of sand on Earth. Keep this in mind as we look at the complexity of optimizing cloud applications.

The Google Online Boutique is a reference application that Google built in order to demonstrate the key concepts of microservice architectures and help train engineers on cloud-native applications. It's an open-source application. It consists of 11 different microservices, each taking a different function of the application. And it is mostly used to demonstrate best practices and understand how to operate and run applications in the cloud.

When you look at deploying that application, how to deploy this application in production and tune it, and the complexity of that tuning process, let's look at the number of variations, number of parameters that need to be tuned. There are 11 services. Each of these services has at least two tunable parameters: the CPU that will be allocated to it, the resource that will be given to it, and the memory that will be given to it. And each of the 11 services needs a specific value of the CPU and memory resource assigned to it.

So there are 22 such configuration values. If we go with just eight possible values for each of them, although this is conservative, there can be more, then we will see that there's 8 to the power of 22, or 75 quintillion different configurations that this application can take, and only one of them is optimal.

Now, you will notice this is 10 times more different configurations than there are grains of sand on Earth. So how would you come up with the right configuration? How do we find the right configuration? Our answer is machine learning. This is the way to get to it.

And to demonstrate this, what we did is we connected our continuous optimization service to the Online Boutique. We took the Online Boutique from GitHub repo as it is published as an open-source application and ran it on a Kubernetes cluster. We looked at it, at its performance, and in how much resources, what footprint it takes, and it was pretty straightforward to see that it was not running optimally.

So we connected it to our continuous optimization as a service system. And these are the results. I'm going to give you the results that we got. We saw an 80% reduction in cloud cost, so you actually need to pay only about 20% of the cost for it. And at the same time, we were able to increase its performance. The result of that optimization is that after it's optimized, you can get eight times more transactions per dollar in running this application. And we did this with about a 20-minute setup. One of our engineers, it took approximately two days for our system to find that unique optimal configuration among the 75 quintillion.

We did this with machine learning. Why? Why do we need machine learning for this? Well, let's look at how we would do it differently, how you would do it with, let's say, brute force. If it takes you one second to try one of the configurations, it will still take 2,300 trillion years to go through the full configuration space, which, even if in your business you can afford to wait that long, you can't do it because the sun will go supernova in only 5 billion years. So I hope this gives you an understanding of the depth and the complexity of the problem. And even if you go with more advanced algorithm, like you say, you do binary search, the scale of the problem is so big that it cannot be solved through traditional means.

Where I'm taking you with this is that tuning cloud workloads manually just is not feasible. Why? How are they different? Or how is a cloud-native application different from traditional applications in which manual tuning was done? Well, the main thing is DevOps. The concept of DevOps brings tremendous velocity. And this is a benefit that we specifically look for in DevOps. That's why we do DevOps.

That brings rapid change in your application on a weekly and daily basis, and even hourly basis. So your code changes constantly. Your application runs on multiple platforms. You have two or three different cloud providers. You have vast configuration space. And the example that we just gave with only 22 parameters, with only two parameters per service, was actually very basic. The moment you add Java or any other sort of middleware, the number of parameters per service explodes. It goes from 20 to 200 parameters per service. So there's a tremendous amount of complexity in that configuration space.

While doing traditional manual performance optimization is something that is reactive. It applies to the configuration and allows you to spend several weeks trying to tune it to the configuration and the code that was there two weeks ago or eight weeks ago. Most teams don't even have a way to measure efficiency, of how efficiently you are running. What is your coefficient of efficiency?

And then the typical solution is that applications are over-provisioned. That is how most applications run today, is they're just being tremendously over-provisioned in order to compensate for not being able to do tuning on the scale that is needed. And the result is predictable. You have massive waste of resources, performance that definitely needs improvement, performance needs to get better, and very frequently, lower availability of the application as a result of performance and under-provisioned resources.

Let's zoom in a little further and get a little bit more concrete. When we look at how an application is developed, starting with the definition, the product owner defines the application. Developers build the code and commit it to version control, then use a good CI/CD pipeline to go through build, test, integration, get through the release workflow through dev, staging, and production, and put the application in production.

The result, very frequently, is the performance in production is not adequate. There's performance lag. It's not working as well as it should.

So what are the best practices? Best practices is add real user performance monitoring, seeing what your customers actually experience as performance. Your team can build synthetic load generators and create various load profiles to test your application with, and then you use sophisticated application performance monitoring systems to understand how your code works, where the bottlenecks are, and try to improve that performance.

That is a very long process, so what happens in the meantime is you over-provision. You add a lot more resources so that the application works and works today and delivers good quality and behavior to the customers. Then as that process of application performance management continues, and you have a tiger team that works to solve the particular performance problems that show up, you feed that back. There's a lot of learning about your application, a lot of guesswork, and a lot of trial and error that are involved to feed that back into the configuration and test.

And then in some cases, that information can go all the way to the beginning of the process and inform the developers as to what they could have done differently three months ago. So this process is beginning to reduce velocity and to impact velocity, and it's just not working very efficiently.

Why? Because it is retroactive. It's looking backwards. This is like driving a car on the freeway by looking in the rear-view mirror.

Now, let's add continuous optimization as a service and see how it changes this process. So the setup is the same. You have exactly the same initial product development process, integration process, CI/CD pipeline. The main difference is now that you add continuous optimization as a service into your production system. The result of this is that your application, as it runs in production, is being constantly tuned for the current load profile using the current application behavior. So today's release or yesterday's release is the one that is being tuned on the load that it is experiencing at the moment. So the changes in configuration and that tuning and cost adjustment is practically real-time. It happens on the application that's running today.

In addition, you can take that process and shift it left, bring it also to the development team, so that the developers can experience the performance behavior and take it into account in development. That is optional, but it is an improvement that you can do to the development process.

And the results are predictable. With autonomous workload tuning, you get higher performance for your application, you get significantly lower cloud costs, and many of our customers report significantly better availability.

So what is continuous optimization as a service? Let's try to define it. What it does is it maximizes performance and efficiency of cloud workloads, and it does that by using machine learning to optimize these workloads continuously.

But let's dig in a little bit deeper. What does continuous mean? When does it work? How does it do this optimization? Continuous optimization can be applied to an individual application, say you have a monolith or several large applications, or your company has a service delivery platform that has thousands of applications. And continuous optimization can be applied in both of these cases.

What does continuous mean? So continuous tuning triggers that process on every code release. So every time there is a new release coming out and being deployed, tuning is being performed.

Load profile changes. If the mix of requests that are coming to your application changes, this can serve as a trigger to re-optimize because resources that your application needs to perform effectively may change as a result of that.

And then anytime when you use cloud providers that let you run your application or even just changing an instance type, any sort of infrastructure change.

The result of doing that continuously is also continuously delivering value. As we talked before, the value is being higher and more consistent performance, improved availability, and significantly lower cloud costs.

How is that different? Let's look at how continuous optimization is different from what generally cloud optimization or cloud cost management as it has been used in the last few years is. When you look at cloud optimization 1.0, it started with cloud cost governance tools. Where are you spending money? How much are you spending? Can you break them down by department, by application, by region? And that analysis allows you to understand better how you're spending money on cloud applications. But it doesn't necessarily help you change that, improve that ratio, or at all touch performance.

The next phase of this is tools. Several tools and companies exist that can make suggestions. They say, well, you are obviously underutilizing this particular virtual machine. Maybe you can switch that to a lower instance type. But notice that it just gives you a suggestion. It says, maybe you should try that. It doesn't do it for you. You still have to decide when you want to do it, like pick a time, allocate a resource to do that, and also evaluate whether that proposed change affects performance and how it affects performance, not only at the time when you are testing it, but also how it affects performance across your typical cycle of load, which may vary day to day and week to week.

What continuous optimization does differently is first, it's continuous. It just happens all the time. There's no phases. It runs every day, every hour. The second important thing is it's autonomous. When it needs to make a change, when there is a better way to run this application, continuous optimization applies it autonomously. There's no need for anybody to look at this and say, well, I'm not sure whether this is going to make it better or worse. Let me try that. The continuous optimization system tries that in a contained, sandboxed environment, and then it promotes that to the application as a whole.

So the impact is always there. Your application is always working at its best without requiring any manual hand-holding that your engineers have to keep an eye on it.

With continuous optimization as a service, you can innovate more efficiently. You can outperform your competition by having your application being well-tuned and having better end-user performance. You outsmart your competition because you can have your engineering team work on adding new features instead of doing trial-and-error configuration changes, and you can outclass them by using your resources more efficiently.

Thank you for attending our session. If you want to learn how to use continuous optimization as a service for your cloud applications or your service delivery platform, please visit us at the Opsani booth. Thank you.