Seven Ways to Implement Standardized Performance Engineering

Hello and welcome to our virtual event today. This is Aaron Goldberg, and I am a contributing editor for IDG. I would like to be the first to welcome you to our webinar.

Today we are going to focus on how to ensure performance and reliability in today's more complex IT environment. Today's event is titled Seven Ways to Implement Standardized Performance Engineering because, much like every other process in IT, we cannot do things on an ad hoc or inconsistent basis; that will just create problems in the future.

The modern application environment has become increasingly complex with microservices-based architectures, multi-cloud deployments, and extensive integrations that make it difficult to measure application performance accurately. The typical enterprise only has a handful of experts that can reliably measure how full systems perform. During this webinar, we will show strategies that can help you develop a performance approach that scales as your application development and delivery practice matures.

As you know, this is a topic that has some technical chops and a little bit of depth. I am thrilled to have with us today an expert speaker who is going to help us understand better how we can accomplish this goal. That is Kristen Webb. She is director of product marketing at Tricentis NeoLoad.

Before we begin and get into the meat of the matter, I would like to tell you and our audience how you can get more value from today's event. First, go to the resources area and download a copy of today's slide deck. It is a PDF, and I think it is actually best practice to take a copy of this. You might ask me why, and I would say there are three reasons. First, it is a great place to take notes as we go along. It is a persistent reminder of this event that you can use in weeks and months to come as you want to refer back to it for more specifics. Finally, perhaps most importantly, it is something you can share with your peers and your organization so they get value from today's event also.

I would like to thank our sponsor Tricentis for making this event possible. With that, I would like to turn it over to Kristen, and we can get going.

Thanks so much. It is a pleasure to be here today. I am really looking forward to talking to you guys about the evolution of performance engineering. That is where we are going to start, and then I will get into why it should be an organization-wide priority, how we can help address the scalability challenge, and some steps to consider for implementing a standardized performance engineering approach.

Twenty years ago technology seemed complex, but we had no idea the explosion of technology that was about to happen. It used to be that we had a few standardized frameworks that we used for development of applications and websites and just a couple deployment models. Then over time we seemed to love virtualization, mobile application development frameworks, and APIs. Agile has just taken off, and so have new architectures based on microservices and SaaS. Here we are today talking about automation and containers and Kubernetes and how we can do everything now in the cloud. That is a lot of change over a short period of time that all companies are trying to keep up with, because now all companies are really software companies. In order to keep up with this complexity and not lose speed, in fact, we have to move faster than ever in this more elastic environment. We need to keep pace in terms of our performance testing. There are a lot of things to consider about that that I am going to talk about today.

For development teams, they are trying to keep up with all of the enterprise requirements that cover everything from AI, cloud-first initiatives, digital transformation, DevOps, virtualization, and security. For performance engineering, they need to keep pace with all these initiatives and make sure they are hitting the highest-risk areas that they need to touch so that what they are bringing to market is reliable. How do they do this? These are a lot of competing demands, and the stakes are very high.

On top of that, we have all these dichotomous application and deployment models now, and they require a new way to test. Waterfall and Agile and DevOps: most enterprises have a mix. The customers I have talked to still need to maintain certain projects under a waterfall environment where they are doing primarily manual testing over the course of weeks for their performance testing on their monolithic applications. That is something they are going to maintain for a while. But as they move to Agile and DevOps, the process for performance testing over the course of a few weeks does not really scale when you have releases going out minutes, hours, days, weeks, and months at a time. They need to really consider how they can scale their approach to keep up with that DevOps environment, to be able to test microservices as they come off, to be able to scale their performance engineering as those microservices touch many different applications and ultimately become monolithic in themselves.

All this is on top of hybrid environments: some of this is on-prem, some of it is going to be in the cloud, and they need to be able to manage both deployment models and run their testing requirements regardless of where those applications live.

All this is typically and traditionally done by centralized center-of-excellence teams for performance engineering. What is happening with DevOps is that a lot of the team is becoming more distributed throughout the organization. Instead of having one group who has the expertise, they now need to be able to move to more of a self-service, self-managed performance testing approach that not only scales in terms of frequency, but also in terms of skills across the organization.

We did a survey with Sogeti. It is called the State of Performance Engineering, and you can find it on the Tricentis website. The results of our survey show that businesses have a variety of approaches to organizing their performance testing. Thirty-seven percent predominantly use a specialized quality engineering team, about a third have a collaborative approach, 17% work autonomously, and 15% outsource their work.

Who undertakes the performance testing for these companies seems to be in flux and likely to change in the next few years. We believe there are two tectonic movements affecting performance testing: the adoption of Agile development practices and the move to DevOps. As people are moving to these approaches, they are expected to be more self-directed and work as peers and teams and participate together and have a higher rate of collaboration. That means it is not just the performance engineers participating in the performance testing process and in the results. It is all different types of DevOps and operations and development and engineering teams that go beyond just performance testing.

But scale is not just about load. When you hear the word scale and performance together, you probably think about putting intense load on an application to make sure it can handle hundreds of thousands of concurrent users under various scenarios. That is, of course, critical in performance testing, but with DevOps the meaning of scale shifted. How does performance testing that typically takes place with releases that take hours -- how can you go from just a couple of tests per month to hundreds and still give enterprises the credibility and predictability and confidence that they are used to? That is also scale.

Performance is not just about load either, and now it affects your business. If an application does not load within four seconds, you are likely to lose 40% of your customers, and they are unlikely to return. That has real negative impact on the business from a brand perspective, from a competitive perspective, and value. It is very difficult to get those customers back. This also counts for internal employee applications that help run enterprises' business. If they are not going to have a good experience using their applications, the adoption of those applications is going to be less likely and the satisfaction of the employees is going to go down. Having performance built into development is a competitive advantage.

Performance testing is becoming an organizational priority. This is the survey that we did with Capgemini. Over 72% are saying that performance goals and metrics are shared with the business users in their organization. Sixty-nine percent say that application performance is an integral part of their brand equity and DNA as a company. Sixty percent say that customer-facing application performance is a driving force to ensure revenue, competitive advantage, and customer acquisition and retention. Seventy percent say internal application performance is a driving force to ensure employee efficiency and productivity. And 60% say performance engineering accountability is at the project level only. That last data point is interesting because what it means is while the business is equipped to say this is our plan and this is an important priority for us, to make performance outcomes and SLOs part of what we expect going into our planning, some of the teams are still catching up. Today we will talk more about how we can help that happen.

Embedded into a DevOps initiative, a low-latency core team of specialist performance engineers can help multiple development teams concurrently to focus on performance. They can help infuse good engineering practice through multiple teams. These teams help with conducting things that are outside their direct ownership, from point assessment to continuous integration.

What I like about this quote is that it talks to the point that in a microservices-architected application, and in a distributed computing world, one bug and one bottleneck in a microservice can have a multiplier effect on your system. You have applications where maybe the infrastructure is located in Europe, but the front end of the application is being accessed in California. How do you know, when those pieces and parts interact, what the bottleneck will be? We need to go shift left and make sure that we are bringing in the development and they are able to account for, at the API level, what kind of considerations they need to make if an API is going to be hit, and what endpoint considerations they need to make early in the design phase, so that as the microservices application is becoming a monolith, it is being built from the ground up in a way that is going to perform.

The answer is to scale the performance engineering approach and not the people. We will talk more about how many organizations rely on a handful of dedicated performance experts, as you have in a waterfall environment. But with the acquisition of a DevOps environment, we do not want testing to become a bottleneck. So far, testing is the number one impediment to faster releases. What we are going to talk about now is how you can scale the approach so that you do not have to scale the people, but you can also create more of an autonomous group of non-experts that are going to be able to participate in the process.

Aaron asks: Kristen, what are your recommendations for organizations looking to get started with a standardized performance engineering approach?

Yes. I brought some thoughts on that, and I am going to talk about the seven principles or considerations to make for adopting a standardized approach. Here is what I will go over in detail on separate slides: centralizing performance metrics; making things easy for experts and non-experts; repurposing functional test assets; standardizing the approach across all use cases; choosing a tool that is ready for your organization; taking an agnostic approach to performance test automation; and then we will talk a little bit about what it means to be cloud native.

Enterprise-wide performance engineering is most effective when there is deep collaboration. An approach that makes it easy for various teams to work together enables performance expertise to scale without adding more experts. This collaboration manifests itself in two ways. One is centralizing the metrics. Developers, performance engineers, and business analysts work together to define measurable performance expectations as service level objectives, SLOs, and ensure that everyone is measuring performance consistently and getting apples-to-apples results.

The second part is distributed results. Centralized performance experts take on a more enabler role. Instead of doing everything themselves, they can create the building blocks to empower distributed teams of non-experts to test at the pace of development.

To make things easy for experts and non-experts: for different teams to use the same performance engineering approach, testing must be easy. Otherwise, everybody will revert back to doing their old thing and their old familiar way. Non-experts simply will not use a testing tool that requires coding skills or other specialized know-how. Low-code and no-code approaches that leverage an intuitive drag-and-drop, point-and-click GUI have proved most adoptable by non-experts. Often, though, DevOps teams prefer to use performance tests as code within a command line interface or YAML files in their day-to-day integrated development environment. Both options should be available so that the approach to performance engineering can be scaled across the organization and everyone can be looking at the same screens regardless of what they are testing and what these cases are.

Three: reuse functional test assets for performance smoke tests. In development, you have your sprints and your releases, but as you have check-ins, you are building your unit tests. What we can do between qTest and NeoLoad is take those unit tests, those functional tests, and convert them into performance tests. This creates a really good early shift-left smoke performance test that you can do on a continuous basis by using the very test cases that you already have. Of course, this does not replace all of continuous performance testing, but it gives you early sanity checks on whether this code is going to perform with a small amount of load, to air out whatever bugs might be low-hanging fruit to catch early and often so that they do not become big issues more expensive to fix later on and also more difficult to fix.

Four: standardize the approach across all use cases and evolving test requirements. To realize a standardized performance approach, you really need to settle on a single solution that is going to be able to cover everything from your microservices, your single-page applications on mobile, Internet of Things applications, or your desktop apps for enterprise packaged apps like SAP, Oracle, or Citrix, virtualized desktop, and your Salesforce instances. Of course, any of these can be run either on premise or in the cloud or in a hybrid environment. You want to make sure that you are covered so that every team and every app and every technology can be performance tested. Leading enterprises have successfully implemented this standardized approach, and while the specifics vary, what they all have in common is that their approach meets the requirements listed here and is easily adopted by experts and non-experts alike with a high degree of automation, which we are going to talk about more.

Five: choose a well-integrated performance solution that works with your tech stack. This is an example of what a tech stack could look like, and oftentimes this is what enterprises have in their tech stack. Of course, it can look different depending on the organization. For most companies, they do have functional test tools they use, and you want a performance testing tool that is going to be able to repurpose and integrate with those functional testing tools so that you can repurpose the test assets you have today and save time from grunt work that you can be using for higher-value work like analysis or getting to those go/no-go decisions within an automated environment.

Speaking of automated environments, you also want to make sure that you are going to be able to integrate out of the box without having to do a lot of hand coding with continuous integration tools. Most companies today have a myriad of CI pipeline types. We want to make sure that the tool that you use for performance testing has support in an agnostic way across different types of pipelines.

Also, application performance monitoring is really very hot. Knowing what is happening in production is always going to be important. As you shift right and you bring that shift-right data into your performance engineering tools, you will be able to see how things in production change on the fly based on changing user conditions. You can take that data and update your SLOs dynamically, and also update your autoscaling systems dynamically as you realize that you might not need as much infrastructure or you might need to scale up your infrastructure.

Version controlling is the way development teams work today, and it is very important to integrate with them and manage your test cases and all your test assets in the same way that you manage all of your assets for your development. SAP is one of the largest applications in the enterprise space for how businesses run. SAP can be the backbone of corporations and how they run their business. You want to make sure that there is excellent support for testing SAP, and then also that you still have the option to connect with any other tools in your toolbox. They can be anything from business tools to how you manage your notifications or how you manage processing your data. That is going to require a performance engineering approach that has openness to it, whether that is an open API and open SDK that helps you create a performance testing approach that fits into your existing stack today and your process.

Six: taking a CI pipeline-agnostic approach to performance test automation is important. We want to be able to move away from manual approaches into an automated approach for performance testing because, one, it is faster and is going to be able to keep pace, and two, there is going to be less human error. Once you have built out your pipeline approach, you know that you are going to be able to keep pace, but you also want to be able to trust the results. Oftentimes teams have super-fast development cycles, but they still need an approach that can seamlessly switch between pipeline types across cloud and on-premise CI tools, container orchestrators, and public and private cloud operators. You want an approach that does not require rewriting test scripts and settings and automation commands as your environment dynamically changes. This is really important. As we talked about earlier today, performance testing teams are supporting the development of business applications that run the gamut throughout the organization, and so do the deployments of those application types. They live in various ways, they are connecting to other services, and they are using different pipeline approaches. You really need to be able to support them all.

Seven: future-proofing your approach. The cloud has become increasingly important, and as we move to more single-page apps and cloud deployment models and more of a fat-client approach, it becomes even more important. No matter where you are in your cloud journey, your approach to performance engineering must be cloud ready. Not only are your applications moving to the cloud, but so are the software developer life cycle, the process, and the tools. Your testing solution should be cloud-vendor agnostic so that the performance and scalability can be measured across different providers like AWS, Azure, and Google. But scalability is not free. You really need to make sure that you can determine when to dynamically scale up and down testing resources on demand, and map changes in load and test when autoscaling might be hiding a performance bottleneck, because autoscaling does not come for free either. While you may have one cloud stack today, that can change, and your approach needs to work across cloud CI tools like AWS CodeBuild, Cloud Build, Microsoft Azure DevOps, and cloud orchestrators like OpenShift, Kubernetes, and the various managed operators like EKS, GKE, and AKS.

Those are the seven considerations for a standardized performance testing approach that we read today. This is just a summary of what we just talked about: centralizing performance metrics and distributing results, having the whole team agree on what the SLOs are ahead of time, and then keeping everybody in the loop as a collaborative unit. You are making things easy for experts and non-experts alike. That is providing a performance approach that provides for using either an as-code or GUI drag-and-drop approach. Making use of your existing functional test assets for early and often performance smoke testing. Standardizing the approach across all use cases and requirements, across all the complex technology stacks that we talked about today and development requirements. Choosing enterprise-stack turnkey-ready tools, making sure that you are going to get those integrations out of the box, they are going to fit into your development process and tools without any issues, and that you are going to be able to use the data from performance testing in any of the tools that you want to. Taking an agnostic approach to performance test automation and being ready to support various pipeline approaches on premise and in the cloud. Finally, really planning for the cloud and thinking cloud native.

Aaron asks: We need the right tools to support this. What are the key capabilities or functionality we should look for in an effective performance testing platform?

Kristen Webb: Thanks for asking. I think it is really important to be looking for these four attributes in your performance testing approach. Cloud ready can mean a lot of things. It means being able to support not only the testing of a cloud migration for your application, but also supporting performance testing with CI pipelines in the cloud, so doing your actual entire development and deployment process in the cloud. Of course, you could have a hybrid environment where you are doing the development for your applications on premise, but you are deploying them in the cloud. You need a solution that is going to be able to support all those different types of scenarios and all of the cloud technology integrations that I showed today.

The second is enterprise readiness. Here, what I mean is being able to support all the different packaged applications, protocols, HTTP, everything to single-page application testing and browser-based testing and package application testing, making sure that all of the use cases you need to get to market are going to be reliable through one performance testing approach, that it is going to be covered and you are going to be supported.

Three, that you are stack ready. Everyone in every enterprise has their technology stack. It is becoming more and more distributed today, more complex, more hybrid between on-premise and cloud. Make sure it is going to fit within your existing environment.

Four and finally, DevOps ready. This means everything from having an as-code approach that can very quickly, with a low-code approach, automate your performance testing within CI pipelines and build in an automated go/no-go decision so that when you get a go or no-go, you can rely on that result because you have performance testing with the right SLOs built into your CI pipeline.

If you want to learn more about the approach that we have at Tricentis, we have Tricentis NeoLoad, and that is our continuous performance testing approach, for everything from automating APIs to whole-application monolithic load testing. We would love to have you come and talk to us for a demo or download the trial. We are happy to provide more information. Thank you so much.

Thank you, Kristen. Just a lot of great information and really a comprehensive approach, which I think, as we all know, is what we need today as we have much more diverse infrastructure and a lot of different problems than we had in the past. Unfortunately, it looks like we have come to the end of our allotted time for this webcast.

I would like to thank our sponsor Tricentis, and again my thanks to Kristen, our speaker, but most of all I would like to thank you, our audience. We know you have a busy day, and we appreciate the fact that you made us part of it. For our sponsors and IDG, this is Aaron Goldberg signing off.