00:00:13

Hello everyone. And welcome to this session. Titled intelligent test selection. I'm Tim Gerard product manager at Tricentis for live compare. So today's agenda is all about the two main components that make up intelligent test selection. So the first item is development. All those changes that are happening in your development environment. And for this session, I'll be primarily talking about changes as they relate to an SAP environment. So these would include, um, SAP author changes like support packs and enhancement packs as notes, and also local development changes. So this has changed related to your own local system. Um, your own customizing efforts configuration as well as any custom code that you create or maintain in your own, uh, local SAP development environment. Um, that don't want to take a look at the operation side. And when, when I refer to operations here, I'm primarily talking about the activity in an SAP production environment, um, mostly usage.

00:01:28

So what objects, uh, or capabilities are used, um, first of all, what's available in SAP in terms of what you have your transactions for your, your web apps, executable programs, uh, API APIs. And then the third part is all about the intelligent test selection, connecting the dots. So taking, uh, and examining the changes that are happening in an SAP development environment and see how they relate to what we actually use in production. Right? So we have the best test candidates to test in our QA environment when we try to invoke those changing objects in development. So let's get started with the development piece of the puzzle here. So there's change going on. And the first question about change is, well, what do we test now? Like I said, before, changes can come by SAP or our own local developers and configurators. So when there is this change event, you know, we all have, I guess, an inkling of what kind of unit tests would be, uh, acceptable to do, you know, invoke that, that change.

00:02:39

But you know, all these objects in SAP are really connected, right? So what else is impacted and is there, are there other objects on our Tash radar that should be considered? So whatever the, the change event is, that's the key question, right? What do we test the first step in intelligent test is to examine the change, uh, and for today's demonstration, I'll be using the Tricentis life compare tool to not own the, uh, analyze changes in development, but also usage in production. So this intelligent test selection tool, we'll, we'll take a look in development, um, analyze and identify all of those changes, uh, and on the right-hand side, that usage data. So those used SAP capabilities, but this is what a good intelligent test selection will do. It will analyze what's changing in dev analyze what you use in production at the operation side, and find a list of core capabilities that are most at risk that you should test.

00:03:45

But this demonstration, not only will we find those capabilities, but we'll relay them well, we'll relate them to your test assets. So your test cases in your, uh, in your testing tools. On the left hand side, I took a look at, uh, one of our systems in the labs, one of our SAP systems, and we applied a bunch of support packs in 2009, 2019. Those support packs contained over a million changed objects. So it was a huge stack that we applied in 2019. And that is not uncommon when I'm, oftentimes when I'm at customer sites, sometimes some of my SAP accounts or our full year behind on support packs. So when they apply that upgrade stack, it really is hundreds of thousands or millions of changed objects. So what I did with, uh, when examining these changes, I said, okay, what, what are the top 10 by type?

00:04:44

And here we see some, um, well, the first one is S M I M I'm not sure if that's very, really an vocable, that's more of like an image icon type thing, but we have our methods, our report programs, we have apps, uh, table, table definitions that is, um, classes, functions and data elements. So the key thing, the key category categorization of these changes is they are , there are mostly all bop components and table definitions. So that's, what's going on when you apply these support stacks, um, SAP, generally, aren't, aren't changing your master or configuration data. Uh, they will from time to time change data related to their standard roles. But when we look at the, the, the bulk of the change, it's very all bop heavy. And when I say all Bob that's, you know, that's SAP code, so it's very code heavy. Now I did a similar analysis for non SAP authored, uh, changes, and that's on the right hand side.

00:05:48

So I took a look at some of our, our most, um, common customers that we're working with, um, a couple of about a year ago. And I said, well, let's, let's take a look at what your own local developers and SIS are, are, are doing in development in your SAP development environment. So we took a year's worth of, of local transports. And what we found was that most of the change is related to data. So are you see tab? You that's, you know, that's data from, from tables, uh, V DATs. Um, and when you see a CGR, those are the, uh, roles, those are security roles. So there's two really separate streams going on here. When we examine, you know, what's, you know, the core changes. Um, and I like to use this, this graphic by James Bach in one of his blogs and I, and I posted the, uh, the blog at the bottom there, the link it's very interesting in this heuristic, it shows the, uh, this round earth, right?

00:06:55

And you have, you have this inverted pie shaped. Um, and I liked, so I took the, the SAP support packs and the customer transports analysis, and I lined it up to this heuristic to just go right under that development, uh, perspective, um, dotted line there. So this has all that activity that's happening underneath the front end, right? These are changes related to your, uh, in the graphic, your, your subsystems and your units. So at this point, we don't have the visibility of from a user perspective, you know, what should we test that are ultimately going to invoke these, um, these changes coming from SAP and from, from the, from your own custom development. Now, as we go further in the presentation, I'll talk about risk, right? And most at risk and, uh, intelligently selecting the best capabilities that interact with the change. Now, all of those capabilities that I will refer to are all from the user perspective.

00:08:01

So on this graphic, it's the front end, it's, it's that lake or ocean where there's the user input in the mountains and in the field. And then you can see the production data kind of flowing through, uh, both the front end and the Sabac and the sub system layers. And, uh, we'll, we'll relate some real life, um, uh, examples to those scenarios there. But I, I just kind of like this graphic to sh to first tee, when you talk about the development piece, kind of sit more kind of see from this heuristic, you know, where it all relates. Um, especially as we talked, we, we discuss the next topic, which is more on the operation side. So for, for SAP production, we're gonna look at all of the used objects that sit on that front end. So whether it's data dictionary, standard, or custom code, all changes, uh, relate, or kind of bubble up to that testable surface area. So I'll use that term throughout this presentation, testable surface area. And when I mentioned that, what I'm referring to are the transactions SAP transactions, the executable programs, uh, web apps, like Fiori and API APIs,

00:09:20

And bringing that heuristic back into the picture here, these all sit on top, right? That use your, uh, perspective. Um, so there's a lot of transactions in SAP. There are a lot of programs and, and fury, web apps and tiles that are available. Uh, what's important from an intelligent test. Um, selection perspective is to figure out the ones that you, you actually use that support, that support business, critical functionality, and ultimately that interact with the changing components below

00:09:55

That brings us to this, this next topic. And, and, and before we get into that, let's, let's take a, a break from the, the slides here, and I'll show you a quick demo. Um, so here's live compare I'm, I'm actually on the Tricentis live compare tool. And to kinda give you some real life examples of, of what we're discussing here. Uh, I built this really small workflow and these workflows sit on the back end. Uh, most customers don't really see them. Everything is automated when we intelligently tests, uh, select test and, and, uh, uh, things that you should look for in any given change event. But it's all, it's all, it's good from time to time to kind of see, you know, kind of peak under the, uh, behind the scenes to see what these, these intelligent test selection tools are actually doing. So I'm going to run this workflow, uh, this, this, uh, action here as collecting all of the, um, transports or in this case support packs from, from 2017. So these are SAP author changes. So in 2017, and the environment that I selected, I applied eight support packs. Well, it's like a, a more recent a year let's do last year.

00:11:15

And here I can select any of these SAP systems development systems that I'm interested in, and I'm going to stay in the system called SAP 43, 1 of my development systems. And the goal here is to find how many SAP changes or, or support packs did we apply to the system. So now there's a much bigger, um, application of support packs here. So let's get an idea of how many objects are contained in this, uh, 142 support stack. So let's bring out another action to do that. Okay. So the results show that there are 1.1 over 1.1 million changed objects from those 142 support packs. Now taking a look at these changes, here's where we'll find changes related to security, um, as well as all of those, uh, components Like, like function groups and so on. Okay. So what about our local development? Let's take a same, you know, from that same time frame, 2019, uh, let's take a look at all the non SAP authored, um, uh, changes.

00:12:39

So in our development box, we had 21 transports. These changes will mostly be related to, to table data. Okay. So we establish what's, what's going on on the development side. Now let's take a look at operations. And in this workflow, we can read the performance history to find all of those capabilities were using in our production system. So in this system, uh, ECP, 100, we're using about 3,100, uh, capabilities. These include executable programs, uh, transactions and Fiore web apps. Uh, we can also find, um, API APIs. Uh IDocs uh, and you may be saying, well, you know, we have more than one production system. You know, we have dozens. Uh, what's great about this particular tool is no matter how many you have, you can retrieve, uh, multiple data sets from, you know, how many, um, however many SAP production systems you have. Uh, so if you have one or two, he just, you just run to, if you have dozens, you just keep on collecting more vacation stats. So let's grab these, the stats And we'll, we'll paste them here And next let's grab all those changing objects. Let's bring that used objects here and are changed objects here. Now it's all about connecting those dots, right? So we want to know what our applications are using so

00:14:38

Uses what Will connect there. And from our changing objects, we want to do that. Where used analysis now, developers in SAP are really familiar with that. We are used analysis. Now it's all about understanding relationship between the two datasets. And we do that by finding object links And then analyzing object links.

00:15:09

So let's see what we got so far. We have our used objects. We can find their dependencies here. We have our changing objects. We can find their dependencies by using this fine object links action. And then we can analyze them to get to the core, what we call most at risk objects, uh, that you should, you should test. Now, getting back to the presentation, let's take a higher look at what we just accomplished there with intelligent test selection. So we went over the use objects at the testable surface area, and we talked about how changing objects can be authored by SAP or by your local developers. Now sometimes, uh, objects that are changed can be way deep. When you look at this heuristic, which means there's, you're less likely to invoke it during your day to day, uh, business supporting scenarios and production. So there has to be a way to intelligently select the best types of tests that most directly interact with, with changes.

00:16:12

Now, the average customer uses about 3000 of these capabilities and about a third are custom. So their own custom transactions or programs or web apps, when we apply an, an SAP upgrade stack, like the one that contained in 1.1 million changed objects, we're going to find that, yeah, there's going to be a lot of impacts to what we use, just because of that massive amounts of change. So we do have a bit of a, a test reduction, right? If were to do a full regression tasks, we're talking about 3000 executable calls, whereas now we got it down to more closer to 2000. It's a little bit of a decrease, but it's really not enough. It's not enough for our customers that want to become more agile. They want to, uh, adapt to change more faster, apply these support packs and upgrades to their systems, their production systems, faster to get to the most updated, uh, SAP versions and take advantage of all those, those new capabilities.

00:17:17

So for our customers who just need that more agile adoption of change, uh, they use our, our most at risk capability algorithm, right? That AI powered, uh, what are the core things I need to test now at this point in the presentation, a lot of folks ask me, well, you know, what's the difference between impacted and most at risk? Because I see we significantly decreased from a, from a full, uh, full regression test perspective, what we need to test, right? It's over 85, under 500 transactions or web apps. This is much more manageable, but how did you get to these numbers? Well, it's, it's all about, you know, going back to this graphic, right? It's all about looking at, from the front end, what at the testable surface area is most directly going to invoke those changes below that's exactly what those most at risk, uh, capabilities do.

00:18:13

And oftentimes SAP will change a really popular object. It could be some kind of function module include our Bobby that's reuse hundreds and hundreds of times by many, many different transactions. So one change can impact hundreds of transactions. And when this happens, live, compare, takes a look at the change and says, okay, well, what transaction at that surface area most directly interacts with this change? Okay. I want to, I want to test it with the most, uh, intelligent scenario, uh, the transaction that directly interacts with the change, and maybe we use it the most in production and it supports business critical function functionality. And so that's what the most at risk test selection does. So at the end of the day, whether you test all of the most at risk capabilities or the impacted you're invoking the same changes, it's just that with the most out of risk capabilities, you're using the most optimal, most efficient transaction transaction set possible to invoke those changes.

00:19:26

So what do we do with those, uh, 479, the most at risk capabilities while we integrate it? Right? We, we integrate it with our testing tools like Tosca and Q test to see of these 479, uh, most at risk capabilities, what tests, uh, relate to them, what tasks contained those most at risk capabilities. And because we don't have 100% test coverage, yet there are going to be some gaps. So we'll find those two. These are most at risk capabilities that when we scan all of the design steps and details of your test plans, uh, we could find no reference of that most at risk capability. So what we do with those gaps is we'll place those in the folder under requirements for our, for Q test or Tosca. So those can be your next, uh, scenarios test scenarios that you build and create, uh, preferably using the ARA tool, the, the Tricentis our recorder. Now, those, those tests that do contain the most at risk capabilities are hits, and those are automatically populated in your execution list. So they could be automated test or manual, but they're there and their execution list and ready to be executed.

00:20:42

So just to recap, so when we find, uh, most our risks capabilities, um, and if they are, are related to test assets or test cases that you have currently, we're going to populate that execution list and all those gaps will go into your requirements. And what's great about this particular test selection tool is we capture all this information and we capture it for, uh, smart insights, uh, going forward. So here's a snapshot of what about seven months here. And let's say we, we implemented a live compare, um, back in, uh, in August. And at that time we didn't have a broad range of, of, of test cases. Okay, we're still, um, getting used to this technology. We're still bringing it on board. So at that time, there were, there were 63 gaps and only 10 hits of the 73 most at risk capabilities that live compare identified.

00:21:39

But take a look at this trend. Those gaps are trending down. In fact, the last month that we ran the tool, there were only 24 gaps. Okay. So we're getting to that point where we're closing all those, uh, test asset gaps. And we're creating the tests that not only are covering the most used transactions, most business critical transactions, but they interact with the most common change events that are occurring from, from SAP and from our own local development. In other words, we're creating has test assets that we need to create that support our most critical business functionality, and that are interacting with change events, both locally and from SAP. When changes come from SAP, like we mentioned earlier, the, uh, the most common change event is op op related. But what about your local changes? Things like configuration, master and security. So if you do a configuration change, right to, uh, uh, you know, to a process, maybe it's in material master or logistics or sales and distribution, you can have dozens or hundreds, hundreds of test scenarios related to, uh, a most Irish transaction, but it's the data component that really helps, helps us filter to the, the best possible test selection.

00:22:59

And when you look at the process, they can process as all those impact the capabilities and think of the data as a further way to filter those capabilities, to make sure that you're using the test that most directly interacts with the data change. Now, this all equates to quality, a better quality of test. For example, a very popular table in sales and distribution is as T vac table. And those are your, your sales document types. And in this example, we're changing a sales document type called TA and in development, we set the, uh, lead time to 99, 99 days. We're in production. It's still wide open. So not only do I want to test my, uh, tests, uh, cases that contain say a, you know, create sales order transaction, we may have dozens or hundreds of these scenarios, but we want to limit them to just the ones that interact with this, uh, sales document type, uh, TA. Now, when we do something like this, it equates to fewer tests runs more accurate testing and a maximum risk coverage because we're focusing our testing efforts, not only on the impacted business process, but the data related the, the, the S the exact scenario that needs to be tested.

00:24:24

And here's a real life scenario where, uh, prior to, uh, looking at the data component, we may have almost a hundred of test cases that interact with the process that was, that was ultimately, uh, impacted. But when we bring in the data component, we can see, we significantly reduce it to just five. We don't have to test a hundred different test cases that relate to creating sales orders, just the five that relate to creating sales orders that use a sales document type TA, because that's the core of what we really need to get at. So data is a huge component, and this particular intelligent test selection tool Tricentis live, compare not only looks at those impacted processes, but it also takes into consideration the data changes.

00:25:13

And then before you release these, these changes and to production, you'll, you'll want to do a test audit, right? So everything you see here in green, that means these are your most at risk capabilities that the test selection tool identified that have been tested and have passed. So when we look back at that execution list and Tosca, we want to make sure all everything's been tested with a pass status, the red indicate, yeah, it's been tested, but it hasn't passed yet. There are still some tweaks that need to be made. Now, you'll see some gray here too, and those grays represent test not run. Okay. So we actually get this data right directly from SAP QA. We look into your QA environment during the test week. We, and we see, or whatever the test cycle is. You get to decide, you know, maybe it's a three-day window. Maybe it's a five day window, but we look and see in SAP, QA has this most at-risk capability even been executed.

00:26:11

Okay. So when you see a gray, that means that, you know, gray or red, right. If you have a failed test or a test or most service capability that hasn't been even invoked yet or executed yet in QA, you know, as a release manager, you're not going to release this change into production, right? Until everything here has a past status, think about your external auditors. They come in every year, they do an audit of your change management control practices, right? What, what did, what is your process for implementing change? And when they say that, when they see that you have kind of control in place that mitigates the risk of untested, uh, uh, change capabilities, finding their way into production, they're going to give you, what's called a better opinion on your change management control practices. You have the control in place to make sure that untested changes don't make their way into production. So it all starts with, uh, an SAP application change. Um, Tricentis live compare can examine the quality of that change, run the smart impact, uh, intelligently select the right tests for Tosca. Then finally do that test audit and make sure there are no defects, zero defects and production.

00:27:29

So I just want to take a moment to talk about the, the latest technologies and the latest releases for this fall and winter for live compare. Uh, we introduced a developer impact analysis, which automatically analyzes your development environment. It picks up any new changes that are going on and finds the best unit tests. So most developers, they may know something that they like as a unit test to invoke whatever change they made. Right. So I think back in one of my first, um, jobs in SAP was an odd query writer, right? So I worked in HR, HR, I S and I would create these, these info sets, right? Think of info sets as a, the data components that, that you build your OB queries against. And I'm this think about if you're not familiar with top queries, they're basically just report programs, right? So I would change these info sets and I had all these reports that are dependent upon those info sets.

00:28:29

So I made sure I change. I would do a unit test on my HR reports that interact with those info sets, but I had no idea how I was messing with other teams, reports that also interact with my info set, right? There were, there were reports and benefits and other groups that also, uh, were dependent upon my info set. So when I changed it, if I removed the field and some other report was dependent upon that field, well, I just, um, I just caused an issue with their reports. So what this intelligent test selection does for unit test is fines. Uh, things that may fall off your radar, you have an API APIs, right? So, you know, when you do make a change, you can self focus on your favorite unit test, but this tool also provides the other best candidates that also interact with the change.

00:29:17

And it prioritizes those candidates by the most at-risk first. So what does intelligent test selection support? 10 times faster testing with accelerated release cycles, 90% risk reduction and proven software quality and 50% lower costs. Reducing the cost with automation, use intelligent test selection to become more agile. I'll apply these support packs faster. Uh, don't wait a full year and apply a full stack. Maybe apply them quarterly, or as they roll out because with an intelligent test selection, we'll find those optimal test cases that are most at risk that you should focus on. So thank you for your time today. Please let me know if you have any questions.