London 2020
Automated Testing on the Mainframe, Yes it Can Be Done
Automated testing for traditional mainframe applications has been a challenge but, over the last few years, significant work has been done to improve the landscape to allow the same type of development.
This session will discuss using OpenShift for z/OS test environments, new open source testing capability Galasa, doing unit testing with code coverage as changes are made and the shift in culture required to do this transformation based on customer use cases.
RR
Rosalind Radcliffe
Distinguished Engineer, Chief Architect for DevOps for Enterprise Systems, IBM
Transcript
Hello, my name is Roslyn Radcliffe, and I'm so glad to be back here with you again at DevOps enterprise summit, London virtual event. Today, I get to talk about automated testing for the mainframe. Yes, it's possible. And we get to have a fun conversation about how you really can do automated testing. So let's get started looking at the test pyramid from Martin Fowler. We see what areas we should be testing and sort of how much testing we do in each one of those areas. And it, when we think about this testing pyramid, it's big at the bottom in order to support that unit testing, that fast feedback. The point of this is we want to make sure that you get that fast feedback back to the developer when they're doing their testing and getting that back to the developer as quickly as possible. So what can I do, unit testing the smallest piece of code I can test.
Let's do that unit testing, make sure I get that perspective. And then what's the next stage of testing. What's that next level of testing that I can do in order to do the testing. And in most worlds, when they look at Java development, you look at things like J unit. When you look at other environments, you look at the fact that normally you do unit testing and then you do automated testing. But when we look at the mainframe space, traditionally, there hasn't been a lot of that automated testing. Why not? Why not? The biggest reason why not is the environments. If I'm sharing a dev environment and sharing a test environment, my data isn't static, it's hard to build those automated tests in order to run this frequently in order to do this kind of repetitive testing. I can't reset the data because everybody's using the environment.
And so we've done a lot of work to help make this better and make this possible. Starting with Z unit, wait, you got J unit for Java. So why not Z unit and let's think about Z unit and a cobalt program. I can build a Z unit for that cobalt program and it will start out all of the program calls. And so in a way, just like I build a J unit, I can do my testing of my individual program without that environment and the way it stubs it out, it actually removes the need for the middleware. So I don't have to deploy. I just ComPilot on Zils and I can run my tests. I can run my unit test on that load module that I could run in production, but I'm doing my early testing first. And then we want to get to the next stage of testing and what's next.
Well, we have this capability called Waze virtual test platform. And with Waze virtual test platform, I can do that next stage of testing. I'm still in the build process. I'm still in that build phase, actually, where I do all my compiles and link it, but I can then run this Waze virtual test platform in order to run that transaction test again without the middleware. So I don't have to deploy it into an environment yet I can run my testing. And then when we get to the next stage of testing, we have all sorts of different tools that may fall into this space. We have tools like Galarza. That is an open source capability that allows you to do end-to-end testing. It has test managers that know all about dos, but also know about things like so many of them. So I can integrate my test.
It is a test framework, similar to J unit, except on a larger scale to allow you to build out those tests and really allow you to get that environment. You have test products like rational test work bench that then allow you to do those test cases that you're building for function, test or performance test on the whole bunch of different tools fitting in this space. And then of course, the end to end testing, I'm looking at a UI testing, things like selenium to do that testing. But if I can focus in that bottom part and get that fast feedback, just like I do in other spaces in the Z space, I can get this continuous integration process. I can get this fast feedback. And here you see what we're trying to do is get this test pipeline. I'm trying to get this ability to do this code and build and get my testing in that process to allow that first loop to actually happen.
And then I move on to the later stages and I've got this provision deploy and test. I can provision a Zoes environment. I can provision it in open shift using Waze sandbox. I can provision it in any cloud environment, using Z development and tests running real dos in the Intel environment. I can run my automated tests and, and all of this helps remove this challenge of those static environments that we once had in the development environment. One of the things I want to do is actually show you a little bit about this. It's, it's fun to talk about it and it's fun to show charts, but how about this time? Let's actually show what we're doing here. Let's start out with a modern development environment. Yep. This is vs code. This is my vs code environment. And in my vs code environment, I'm running the open editor to allow me to do my cobalt programming.
And so here you can see on using the IBM Z open editor it's available in the marketplace, go download and you can edit cobalt peel one assembler programs JCL and there you can see, there are a few other pieces around here. I've got the user build capability. I've got a debug capability in here. I also have the glass capability so I can do my glasses testing. I also have one other thing in here. I have added the Zoe environment. So I've added the Zoe Explorer into this environment to allow me to interact with my cos system, from my familiar interface. So when I'm building my source code, when I'm building my test, I can easily interact with that system without having to go to the system. I know some of you might be more familiar when you think about Zoes you might think this is what Zoox is this 32 70 green screen.
And, and yes, that's there still. And if you are familiar and you like ISP, that's fine. That's where I started in IBM development, but I can use a modern editor if I want to, I can use a modern environment so I can get that experience that I want to and allow me to use. The IDE I want to, there is an eclipse editor, there's this there's also an eclipse Che environment. So if you want to, you can run this edit experience in the red hat code, ready workspaces with IBM Waze for red hat code, ready workspaces. So you can run this in a cloud native development environment, sitting alongside the Waze sandbox. You have that full dos environment to do this testing, to allow you to write your code and test it before you ever share it to anyone else. And if we look at this Zoe Explorer, what I'm doing here is connecting to my ZEOs environment to deal with the files that I have to deal with the file system.
I can look at jobs. I can look at the environment and do what I need to do with the system. What I'm using here is the remote system Explorer access to Zils. So it gives me that full function development environment that I need to allow me to connect, do the kinds of things I need to do, allow me to do my user build, allow me to do my different steps. Using the Zoe Explorer, you'll see in this environment I'm integrated with get. So my source code is actually in get in this case, it happens to be in get hub and it's sitting here managed in, get hub, and you can see I've got all of this COBOL code sitting in the, in the get hub environment. Okay. That might seem a little unusual, but how else am I going to do this? If I'm going to do my automated testing, if I'm going to do my processing, why shouldn't I have everything together in one SEM building, how is my source code with my tasks, with a Java code that I'm writing?
That's the part of the application. And with the front end application, all of the pieces can be together in the single SCM. And so I can work together across my teams. Now back to the code, I love doing live demos, but I'm not totally crazy. So when I do a live demo, um, I make my changes in comments. So I can't make a coaching change. And so I make a change and I save that change. And when I save that change, I know I've made a change. Get tells me I've got this change. I would logically do a user build and then make sure my code works before I move on. But in this particular case since well, don't have forever. We want to see this working and I want to kick off this pipeline. I'm going to go ahead and check this in because you know, I know it's going to work.
I did it right. It's a comment. So it's pretty good chance it's gonna work and you don't want to watch me typing really, but here we go, new change for the demo. We're going to commit that and then we're going to push it. And so it's going to flow and be checked into the get environment. What happens when I make a change and push it? My branch I'm pushing it up. Well, it should kick off a pipeline like it would in any other environment. And so let's see what it does. It kicks off a pipeline. It's kicking off a pipeline to run my process. Now, while that's running, we're going to take another look at one other program here. When I said, this is my cobalt program, then I met hitting and you can see this is a, a full cobalt program. And if you're not familiar with COBOL and you don't know what you're supposed to do, it's a full editor.
So you have the full experience to understand how to write whatever you need to write. And if you take a close look at COBOL, it's really just English, move something to something. It's not that hard to understand my full modern editor, but I also want to have a test case. So I want a Z unit associated with this program, and this is what I've done here. This is my Z unit that was created in the using IDZ and generated for me based on record in playback. And so using on the IDC environment, I can create this unit test. Once I have this unit test, I can edit it anywhere. It's cobalt. If I do a cobalt program, I generate a COBOL unit test. It's what makes sense. And you can see this looks like a normal unit test. I'm checking data to make sure that my program worked as expected.
And I can have a series of tests within this one program that get run whenever I change my program. And so I can get that fast feedback and get that, you know, feedback right away, as soon as I make that change. And that's what has happened here, I've done my build. And so I've done my dependency based build, made sure it built the right pieces. And then I ran my unit tests. And so I got that fast feedback to say it was successful. And then I ran this next step. I ran this, this Waze virtual task platform or Waze VTP tasks that allowed me to test not just my program, but my program within the entire flow of that program, that transaction that I'm running. This happens to be, uh, a Kix DB, two transaction. And so I get to test that whole transaction. So if the program changed that I made caused a problem in the flow, I'd find out right away, the system would tell me that what I changed, broke something else in a related program.
I can get that really fast feedback before I've ever deployed it into an environment. And then I take my change and I put it in. Artifactory just like I would, for any other system alongside the Java code, alongside everything else I've got in my environment. I put it in Artifactory so it can then be deployed. And so I then do my deploy and the deploy process puts it into a kicks DB two environment so that I can run my integration tests and this case I'm using glasses to run my integration tests, and it's going to run us at a test, but I've also put something else in here. And I don't always have to run this. I can selectively run this, but I'm doing a recording of the data before turning on the recording, running my integration tests, and then turn off the recording so that I can create a new data file.
So if I change my program significantly, I make new environment calls. I can get new data to use in the build phase. The idea with these tools and this capability is to allow you to do things faster and sooner. And by being able to stub out the middleware entirely, I can run that program as a batch program. I can run it standalone, but I haven't modified the program at all. It's the same load module that you can run in production. You can run in this way. It's a set of new capability to allow you to do that. Using the characteristics of dos. It allows us to do this. And so we can have this early testing and early feedback running in the, in this environment. Even without the middleware, I can run it and debug, I can step through the program. I can get that fast feedback without having to have that full environment. Yes, I'm running on Z O S still, but that dos can be running in my OpenShift environment to allow me to do this so I can do it all in a cloud native way and get that fast feedback and get that understanding of what I've done and how I've built. It.
It's really important that we're facilitating this whole life cycle when we're doing Zoes development and in the same way that we're doing it in the rest of the world, we're just, we're trying to bring Zoes into the rest of the processes. So you have existing processes around selenium, around other testing tools, bringing those into this environment. And in this picture, what we're trying to represent is there were a number of tools that we provide and a number of open source tools that play a part in this pipeline. And in fact, virtually it's your pipeline, whatever pipeline your using you plug in the Zoes environment into it. If you're currently using cucumber, I know people who have set up cucumber to talk to Zils to allow you to front end that capability. And that'd be another place we could look at building a manager for glasses to extend it into that space.
It's an open source tool. You can extend it. Anyone can extend and add additional capability, but it's your open source tools that you're using in this environment alongside the special day capability COBOL, for example, to allow you to go through this process and allow you to get that feedback. But now we think about what I've talked about. I write my code using modern IDE modern experience. I build my Z unit. I can modify my as a unit, I'm doing it at the exact same time. I'm doing my development. And when I do my build, I get that instantaneous feedback. I get that fast information back about whether or not what I did worked and whether or not it broke something else. And realistically, when you've got a big program, what's your biggest worry that you wrote the part of the code that you changed correctly, or is it that you broke the stuff around it?
And it's the stuff around it that, that usually comes into this problem. The other pieces there, those include files that you have in your programs and Cobalt's copy books. It's the data structure that's shared across programs. You got to change a copy book. You're always afraid about recompiling everything, because what else is that going to break? Well, this automated testing, I don't have to worry about making those changes. I can get that fast feedback in the system and building up my integration test. My early integration test using dynamic environments allows me to do this. Experimentation allows me to build up my capability and tested in isolation before I affect anyone else. And so the existing environments that are running on existing Z hardware can be dedicated to those late stage performance integration testing, or late stage in the pipeline. Doesn't mean late stage. As in late in process, it means late in the pipeline.
I can use those static environments for that later stage testing in these dynamic environments, I can provision and provision the data to it that I want. So I can run that automated tasks and I can get that fast feedback and I could keep going around the circle. Now there's no perfect picture. And there never will be a perfect picture. And so monitor will realistically that monitor needs to be happening as I go along the whole process. And so I should be monitoring the system all the way, but we never have a good picture. There's always a question and there's no way to draw this perfectly. So I'm monitoring these systems since it's still running on Z O S in every one of these cases, I can use the existing monitoring tools to get that feedback on how it's performing. How is it really doing? How do I get that? Am I performing the same way I did before? And you can do those like to like comparisons. So yes, if you're running, when you're running your tests on, on ZD and T you're on the sandbox, it's not going to perform like it's going to perform on real Z hardware, but I can do a, like to like comparison. I can see that it performs similarly, and I can get that understanding of how it's doing, and then do the performance test on the real Z hardware in the environment.
You'll notice here I have GitHub. I have get lab. I've put, get in the picture because get is so common as an SCM. And it's the place you store all of the front end artifacts. So I might as well use it for all of the artifacts. I can store my testing artifacts, the new Z unit capability. I can store it in, get alongside my source code. I have all of that information all in one source code manager and it can flow through and it can process all in the same way. Another really important thing about this pipeline and testing have been talking primarily about development, testing, testing, to make sure the code is correct, but you'll notice something else in this environment. I've got AppScan scan in here. I've got security scanning. When we're thinking about this process, security scanning security by design needs to be included from the very beginning and Zoes development is no different than any other platform when it comes to those requirements.
I still need to think about how I write my code. I still need to make sure I'm following good coding practices. I'm following good coding standards. Am I doing the right data validation? And am I running for the scanning to make sure that I'm meeting those requirements? I'm satisfying when I need to. And I'm checking and validating developer can get this feedback. It can be part of the pipeline so I can scan and watch you'll notice here. I don't have any errors. Zero app scan errors found at scan has a set of rules. They may not be the best, but there are a set of rules that look at COBOL, things like SQL injection data validation, whey you're dealing with data in general, all of those kinds of things that you should look at in any environment you should look at when you're paying attention to cobalt or any other language on Zoox, same rules, same process.
It's just as important to remember those security scanning as part of your process and security by design in part of your process, that's all part of the testing that you need to be doing from the very beginning to make sure you're not introducing problems. And I know it might seem a little strange. You don't hear Zoes and testing for security very often because we talk about Zils as a secure platform, because it is designed and built to be secure, but you're still writing code. Doesn't matter where that code is gonna run. You're writing the code. So the code practices need to be the same, no matter what platform or language that you're writing for. It's all fitting together in this process.
Hopefully I've given you a good picture of automated testing on the mainframe and how it's possible as well. It requires a change or requires doing it. It requires implementing change, but it's there and it's possible. You can shift that testing all the way left. You can get that fast feedback. You can get this testing in the build process, just like you can for any other environment, platform or language. Now, one thing I always like to do as part of this is ask you, I'd love to hear your stories about automated testing and how you're doing it. And what do you find most useful? What tools are you using? What do you find more useful, most useful? So you can give me that feedback. If it happens to be open source, let me know if you want it to run on cos as well. And we can look into it.
We've taken a lot of things and worked to get them running on Z O S or just tried them and they work on Zoes. So it's important. Let's look at the appropriate tools and help make sure the Zoes is just another platform. And you can see it as a big server that just happens to be reliable, scalable, and provide those ilities that we all talk about. But from a development perspective, a testing perspective, I can do things just like I do in any other development. And I can work in a cloud native development way, even for traditional mainframe development. Thank you.