00:00:13

Hey there. My name is trace Bannon. I'm a software architect, engineer, dev ops strategist, and a senior principal with the MITRE corporation. Let's bring up some slides. So MITRE is a federally funded research and development corporation chartered by us Congress. And much of my focus is working with the us government, but also with the coalitions partner nations, um, to solve complex problems with software today, I want to share with you what I'm observing as a missing link for DevOps and that's data. So with the help of conferences like does, we are all talking about the challenges of adopting DevOps together. We're waiting through approaches and learning from one another. We're moving towards a more common understanding of what that ops is, and we're creating that common lexicon and we're working together to figure out how to resume, reduce that resistance to change and how to remove that dev versus ops mentality for even learning to reign in our over-focus on tools, even.

00:01:18

So research shows that our quality is not improving at the rates that we expect. I first heard Joe Wessel used this phrase and it resonated with me. Quality is going down the drain with the pipeline. If we are automatically testing, how can quality go down? We're all a unit testing, right? We're all focused on code quality and coverage, right? Well, we're adding in some selenium for good measure. If we are automating testing, why are there still quality escapes? We are all familiar with the infinity loop for dev ops by now, and looking at this, can you spot the problem? Notice that test isn't isolated step, simply shifting testing to the left or right into production. It's not the answer quality and testing a part of every aspect of dev ops. We need to rethink the wording of the infinity loop. Testing and quality are the very fabric of the entire repeating life cycle.

00:02:17

So let's, let's draw it like this. We need to be more specific about each and every phase, step and transition, whatever you want to call those colored chevrons to call it specifically how to amplify testing and quality continuously all. Yes, I have a love hate with that word. It makes so much sense constantly and chronically evaluating and improving. I had an opportunity to meet with Janet Gregory, the coauthor of agile testing and the founder of the agile testing fellowship. She shared a version of the infinity loop that I can really get behind the term. Continuous testing has been hijacked. It was intended to me focusing on testing and quality across the entire effort. So let's use the term holistic testing to encompass all quality focus during discovery plan and understand, identify risks, and test assumptions and build prototypes during build and deploy instrument. The code automate tests include stories and features, infrastructure testing, pipeline, and quality attributes for the whole system. Moving forward to release, observe, and learn, test and production and observe how customers are using the product and how we should adapt. Well, now that we have a focus on quality throughout, let's talk about other gaps that can reduce the effectiveness of testing.

00:03:48

We know about some of the more obvious gaps about lack of cohesive test planning about skills gaps, and a need for training about overly fragmented tests. We know that software is still tightly coupled, and that there was a flawed belief that architectural design just emerges. Spoiler alert. It doesn't well, given these known nuns and given that industry is starting to address them. We're still seeing failing tests according to a 2020 report by undo corporation and the Cambridge judge business school, failing tests, cost enterprise software market $61 billion per year with nearly 620 million developer hours wasted on debugging. Did you hear that 620 million hours? It's amount of staggering, but why, why is debugging so difficult? Because we can't reproduce the defects. I sometimes call these Phantom defects, little bit of humor force. I love this graphic. We have so much available to us, literally decades of test engineering theory on white box testing.

00:05:03

And if you're not familiar with white box testing, it means you have access to the code base and you construct your tests based on what you see in the code base. We have direct access to stakeholders and we have tools and we have tools and we have tools. Well, I'd like to share a story with you for a very large organization. I was an enterprise architect with a team of architects, infrastructure, application, and security all on my team. And we worked across many lines of business. As a shared service capability. The testing group also lived in the same. It shared services organization. It has us, there was a heavy push for automated testing budget, clip the wings of the testing group, and they fell back to muscle memory, manual testing, and it happened late in the cycle. I began to hear of inconsistent, inconsistent test results and errors that couldn't be reproduced Phantom errors.

00:06:05

What we uncovered was that some of the applications had heavy state management and there needed to be refactoring. However, we also found out that the data being used by QA was stale. They didn't reset it after every test cycle and the quality of the test data, what was suspect to, and this really highlights that exercising the code isn't enough for a test to be valid. There has to be repeatability and the data has to be reset manual exploratory testing. Oh, that shouldn't be part of the pipeline. It has to be a synchronous and autonomous essentially lack of adequate test data is eating into the productivity gains and into dev ops investments. We need to know that there's a change on the horizon. So let's work at changing the narrative. The mad dash to automate testing has delivery teams asking, how will we test this? Well, if you're going to stay true to holistic testing, you need to ask a broader set of questions you should ask, well, what should the test accomplish? Why am I going to test this feature or this capability I should ask how I will test it. And then I need to ask what data is needed, the determination of what to test and how to test it. That's a decision of the architects and engineers in the business. The key to effective fruitful testing though, is having an intentional focused on defining, acquiring and managing data for testing.

00:07:40

So what types of data do we need for testing? Pretty big question. If we were together in auditorium, I would have you just yell out at the types unit security performance integration end to end usability and so on. But beyond the types of data, there are myriad considerations overlooked when gathering test data. What's your domain. Um, are there unique requirements, for example, banking, insurance, healthcare, are there regulatory constraints, PII, PCI Phi. Well, if you work with the government, are there data classifications to be concerned about is the entire team aware of the architecture and the requirements and the quality attributes. Now, not everybody knows the term quality attributes. Those are the cross-cutting capabilities, those, those ilities portability scalability and so on sort of nonfunctional as well. Once we have a better idea of the profiles of the data necessary to support holistic testing, who's going to make the data well, often test data is created as an afterthought, uh, and the responsibility while it's loosely defined, it might even not be defined generally tested.

00:08:58

It will be made by some combination of developers, QA, testers, security, testers, and database administrators, DBS, and each of these folks approach it from a very different vantage point. So developers focus on that white box testing. I mentioned earlier, um, where it's based on knowing what the code looks like and their data represents best case. Remember their focus is on writing new features quickly, QA testers often drive their data efforts using spreadsheets and maybe even inherit the developer data, um, when they can, they will swarm to using a set of mass production, data security testers create their data almost exclusively manually right now, including ad hoc approach to it. Now with a focus on dev sec ops, there is a pivot towards an intentional generation of security testing data. Let's talk about the, as though they feed the whole team by bringing production data into a lower environment.

00:09:58

Um, and if we are lucky, they may even apply some masking techniques, but I'm not a fan. I'm not an advocate of depending on production data, look, production data represents an accurate image of data that works well. However, you're only covering 77 to 75% of the scenarios and you're missing those edge cases that cause the headline defects, but most important reason that I found I'm using production data is that we're not masking it and we're not protecting it in non production from hackers, according to the state of database dev ops from 2019, the report by red gate, 65% of companies are using production data for testing. Only 35% of them are masking. Uh, the result is a vast cyber risk footprint. If this is the case, well, where do we get the data from? Well, we should take a blended approach. We need to build borrow.

00:11:06

And by building data involves using manual techniques, um, interacting with the front end, uh, and applying business rules to synthetic data generation tools. And that forms a strong basis for your debt. Your testing suite borrowing data means selective production, data borrowing and masking it. It also includes referencing some prior systems applying an extra load and transform process Neal to buying data from other corporations or data brokers or using publicly available data, or even web scraping can round out your approach to creating a robust suite of test data. So let's say that we have a good basis and we followed the principles for test data acquisition. There are still some stumbling blocks, preserving test data and loading reloading of it to repeat an earlier lesson, if you don't have a valid, repeatable initial state, a test is not valid. Tying this all back to dev ops. Delivering value means delivering quality, delivering quality demands, holistic testing that can only happen with robust test data management, test data management tools and an integration into the value stream and the dev ops pipeline like other DevOps hurdles, applying technology. It's not the challenge and it's not the solution.

00:12:38

You need to be aware of likely obstacles with delivering value. Using modern software practices. First have a test strategy. This does not need to be long and complex, create a living strategy and point folks to it. I touched on data acquisition earlier. Another hurdle teams face are massive use of databases instead of a focus subset. This is unwieldy, it's costly and without a strategy for managing it, it's prone to stagnation. And to leaks, I've been surprised at the number of teams that try to create their own masking tools. There's an explosive growth in data perfecting and masking tools, and those can fill your needs. Now there are budget constraints, but they're being applied to test data creation that needs to be rolled back. It isn't free. So it needs to be included in the planning. I mentioned data sprung and interesting logs are a type of operational data and they're needed for testing as well.

00:13:42

Access to logs is often a sticking point for teams. So I want to share another experience story with you. I was working as a software architect with a massive government agency. One that has very complex test data requirements. The most I've ever seen and almost I've ever experienced. Every public release of software had to have a full production data archived, aligned to policy and law. The agency had put a priority on automating all tests and therefore all data, a centralized team managing the test data was understaffed and they attempted to either use production data sets or create them from scratch many, many, many setbacks from this approach, the delays and failure taught us the value of making risk decisions on what to automate and the value of profiling data and using the information to feed synthetic then generation the QA data team also became strong supporters of self-service data provisioning.

00:14:48

So one state has created, it's gotta be managed, managing test data carries with it, its own set of considerations, deliberate and controlled data access, maintaining test data accuracy, securing the data in non production environments, storing filter criteria, data sets, and taggings against the user stories that they're relevant to and the test cases or the code branch data set versioning is important and data refresh on demand. Now, remember you should delete test data, including the database logs and other files. While there are test data management tools and platforms that can help selecting a TDM means you need to do a trade-off analysis. So each one of these categories has a few different subtopics to think through when we design your test data management platform, you've got data source, diversity, data cloning and verdict and version sensitive data masking and encryption security and access control data generation test data warehouse, open API APIs for integration.

00:15:57

How about that self-service feature. I mentioned before. So I've included what a colleague of mine calls an eye chart. There's a lot of details here. When you download today's presentation, you can check it out in more detail. So let's get back to the dev ops or the dev sec ops pipeline. There can be data friction moving towards quality, first value delivery version, your data sets alongside the code and the configurations reduce unnecessary copies, but enable self-service leverage and integrate API APIs. Think about virtualization allows saving of data after a test is complete. Sometimes you need to come back to it and be able to evaluate that exact state in short cleanup happens and remember logs or data to consider applying and adding performance and load testing into your pipeline. So over the past few years, I've learned that some, that if something is really common sense, I should probably state it out loud as well. So I'm going to give you a couple of things that you shouldn't automate. Don't try to automate a test for an anti automation feature like capture don't focus on infrequent or low risk tests, much like the user experience story I gave you earlier and allow there to be exploratory testing that's manual.

00:17:32

There are many players in the test data management space. Um, here are a few examples. I encourage you to check out these links and to talk to these vendors and to understand what they are using and what they're doing. Go ahead and ask them about lessons learned and what industries that they're focused on. Looking ahead, we are going to see an emphasis on testing and test data management. We're also going to see, uh, an emphasis on those tests, data management environments, I AAC, and automated dataset provisioning, um, allow us to spin up new test environments at parody to production. Digital twins are gaining an importance, especially with some of my sponsors in the U S department of defense. The ability to have full simulation of hardware virtualized gives the ability to react earlier. Now, service virtualization has been around for a number of years. The tools to support mocking can really improve fine grade unit testing as well as help to decoupled brittle integrated testing environments, data virtualization, database virtualization. They're growing as well with technology. Now at the enterprise grade, we've covered a lot today. I'd like to provide you with my contact information so that you can reach out to me. And you can ping. Let's talk more about DevSecOps. Let's talk about software architecture and let's talk about the data. That's that missing link that we need. Thank you.