From Milestones to a Continuous Quality Assurance Flow – Applying DevOps in Industrial Environments

One of the biggest hurdles to implementing continuous delivery and DevOps in industrial environments – dealing with milestones / macroscopic quality gates – can be overcome with a continuous quality assurance approach that applies the "green to green" and "shift left" paradigms not only to software, but also to non-software artifacts.


In an industrial environment, software delivery requires much more than just working software. To formally release a change and deploy it into production requires many further non-software artifacts that cannot be covered by the continuous integration tool chain, no matter how sophisticated it is.


However, using the classical milestone / macroscopic quality gate process is no longer an option, as this limits the deployment frequency to values that are outside the range of the desired target. Therefore, a rethinking of the release process is required.


This presentation provides insights into our experience with several implementations of a new approach to analyzing and tracking non-software quality criteria through the introduction of a continuous conformance concept.


This concept applies the "green to green" and "shift left" paradigms – well established in the software space – also to the non-software artifacts required to release and deploy a software change.


Combined with a continuous delivery pipeline, this results in a continuous quality assurance flow that also enables industrial enterprises to take advantage of the many opportunities and benefits that arise from continuous enhancements of their products and services while maintaining their high quality standards.

PF

Peter Fassbinder

Principal Expert PLM Process Innovation, Siemens AG

Transcript

00:00:14

Welcome everybody. I'm quite happy to be here and to share our experiences on applying DevOps and industrial environments. And the topic I'm gonna talk about from milestones to continuous quality assurance is a very crucial aspect in this context, before I start just quickly, a few words on myself, my name is Peter Fuman. I'm a principal expert for PLM process innovation with Siemens ran consulting. I have considerable experiences more than 50 years in being an agile HR. And now for more than five years, I'm almost exclusively dedicating my time to the topics of continuous delivering DevOps. And especially to the question, how can they be applied in industrial or regulated environment? I'm also quite active in the external community though. I'm happy, always happy to share my experiences on conferences like this.

00:01:14

And when we want to move in this direction company like ours or in other organizations, we really need to, to, to address. What's been shown on this picture, the stop thinking project starts and can continuous value stream change. And this is a significant move. And you can see there's if you really look at where typical industrial organizations coming from and where we want to move towards these more frequent deployments, I mean, a typical industrial organizations coming from a project based approach with projects running anytime between maybe six months, maybe one year, maybe even more up to one and a half years, or maybe longer in between releases of new versions. Now we want to move toward these more frequent deployments. And even if you're not talking about daily updates, but for example, talking about product enhancements with every sprint every two weeks, for example, or maybe even only once per month, I mean, we are always talking about at least an order of magnitude, more frequent than where we're coming from.

00:02:22

And this is a tremendous and significant change. What is also important, especially for the rest of the presentation. Now I'm talking about this continuous well stream, which is shown on the right hand side of the lower era here on the left inside. We have something I called in startup, which means we already have released a minimum market product into the productive system we had had hopefully in successful ensure market launch. And now we are addressing the question and how can we implement frequent deployments, updates enhancements of this minimum market product while it's already in production. And this is the basis that we assume for everything we're gonna present. When you talk about this continuous quality assurance flow and moving in this direction towards the more frequent deployment has already pointed out, has a significant change on a lot of the topics and areas within an industrial or regulatory organization.

00:03:24

This ranges from a tool chain, from product architectures on the table, on the technology side from new and change modified processes, responsibility, uh, organizational potentially changes all the way up to, uh, potentially redefining your motor, your relationship to your customer partners. And what I'm talking about in this presentation on this continuous quality assurance approach is one topic within the processes box here, but it's a very important topic and it's a topic on how can we ensure and move towards a more continuous quality issuance approach in this so-called more continuous world of continuous frequent updates of the life system. And first I would like to show you a few slides on the core idea, the core concept behind this continuous quality assurance that we developed in the last couple of years, and by now have implemented successful in several organizations. And then I gonna dive a bit deeper into, uh, how this can be managed efficiently in practice.

00:04:35

This is a very important slide as this kind of contains the key idea of the continuous quality assurance approach. And it's all based on the queen tore mantra, which I hope you are all familiar with it on the lower part of this picture, labeled continuous integration. That's again, something where I'm pretty sure you're quite familiar with it. It shows basically the shift from a project based approach to a more continuous value stream of, uh, for the software in the past software was somehow created over course of couple of months and variated and validated and eventually data towards the end of the project. You had running software and implemented into other productive system today with a continuous DevOps screen tore approach on the software side. This looks very differently. You, as you already have a running system up in production and you want to implement a very small change on a frequent basis, uh, into the productive system.

00:05:35

So for every change you implement, there are two things you need to do. First thing is of course you need to identify whether it's doing what you're supposed to do, for example, some functional testing, but what is much more important is actually to ensure that this change does not break the running software, that the software system that is in production after you integrated the change is still up and running is still cream. This is the cream tore approach. So this has the uppermost priority in this continuous integration, continuous delivery approach. And this is something where of called from the past. K tremendous focus has been made on the tool chain on test automation on the continuous delivery pipeline that ideally at the push of a button, uh, every developer, every team with every culture that can immediately identify and verify whether the overall system is still cream.

00:06:30

So this is great. And this is of course, something you also need in an industrial environment, but it's not sufficient. Even if you have a perfectly running fully automated tool chain and with an instant at the push of a button, you can identify whether, uh, software system is still running after your limiters of change. You are not in a position to really deploy this into the practice system. Why? Because if you want to deploy software in an industrial environment, you need to deliver a lot more than function software. It's not sufficient to have software alone, even if it's flawless with the highest quality you need additional things to deliver. And I like to use the term non-software artifact for it. And this comes from a variety of solves and requirements. First of all, there might be some, for example, user documentation that the end user need, maybe some frequently asked questions, maybe some online help.

00:07:31

You could also potentially have some, uh, customer service that require some artifacts, some installation guidelines and so on. And you also have some documents or artifacts that you really require for yourself or by regulations. We talking here about maybe patent applications or, uh, export control, open source, risk assessment and all things like that. So there's a lot of, as I pointed out, I like to call them non-software artifact that also need to be available. They need to be in a valid state that is actually matching the software you want to implement. And this needs to be available, verified, validated. At the point in time, you want to deliver a soft and deployed into the productive system, and it's mandatory to have them. You're not allowed to deploy the software without those artifacts. And if you're looking now again at the project based approach, the mechanism to ensure that before you deploy release something, after a large project was exactly the so-called milestone or quality gate approach.

00:08:36

And, uh, now we want to move towards a continuous value stream. Uh, it's not, uh, possible anymore to apply this milestone quality, live this my and quality, uh, approach with every deployment because it's just too, too heavy. It wouldn't be feasible. It wouldn't be efficient to really manage this on this, uh, with approach. So you really need to redefine the way you look at this quality addressing those topics and how you release it. And what we now applied is actually, we said, we're gonna take this screen tube digital approach, which has proven to be very successful to ensure the quality of the software. And we also take it in the same way. And we in ensure by this quality of the nons of artifacts and we label this continuous conformance, and it basically includes two things. First of all, you need to make sure, uh, that you identify which of the nons-software artifacts is affected by a certain change.

00:09:34

You want to implement, meaning you need to identify which of the non-software artifacts require an update. And then you need to ensure that this update is completed and done before you deploy it into production. And this is again, shown on a high level on the yellow part of this picture. What we want to achieve is a continuous, or you could save ER, deployment into the productive system, and we need two equally important streams to ensure this. We need the continuous integration stream that ensures the quality of the software as such itself. But in addition, we also need the continuous conformance team that ensures the quality of all corresponding non-software artifacts and pose are equally important personally, to be clean and only pose are green for a specific change on new feature. We can implement this into the productive system. And, uh, an interesting thing in here, which is shown on blue is that when we started implement this, this approach, we, uh, uh, identified that quite a number of so-called quality car hair, which were in the past covered by these milestone checklists.

00:10:49

They do not need to be, uh, tied to this high frequency, continuous deployment schema. Examples are for example, like a test strategy or, uh, like, uh, um, um, marketing flyers and things like that. Of course, they also need to be kept up to date. They need to be addressed. They need to be managed, but they don't need really to be tied to these individual deployments decisions. And this has two effects. First of all, it makes a continuous conformance stream, the high frequency stream, the yellow part in this picture, leaner, they are less artifacts we need to take care of in this high frequency activities. But on the other hand, we, we must ensure that we don't forget those artifacts. We need to define appropriate mechanism to handle them more on an organization or program level. And all this taken together is what we call the continuous quality assurance approach.

00:11:47

Now, for the rest of the presentation, I will dive deeper into this continuous conformance stream because this is what is really the noble part in our approach. And if we dive a little bit deeper into it, another thing we cover that it's not sufficient to only apply the queen tore mantra. We also need to think about applying the shift left mantra also to these non-software artifacts. Why is this the case? I can explain you briefly. Initially we thought we want to achieve these frequent deployments. We have, for example, two week sprint, we have a continuous integration, two chain already implemented multiple times per day. This is running. So we have a lot of small cycles where we ensure the quality of the software. And by the end of a sprint, we can, from a software point of view, be a position to deploy into the practice system.

00:12:40

Now, our initial idea was thought was, well, now if you have potentially two weeks time at the beginning of the sprint, we can identify which of the non-software artifacts are affected by the changes we want to deploy. And then we have two weeks time to address them and to update the corresponding non-software artifact that are affected. And then by the end of the sprint, we have them ready, but we very quickly realize that this is not, uh, possible for two reasons. First of all, some of these, uh, non of artifacts, for example, patent applications, if they require an update, this takes a long time, more than a sprint typically. So it means we need to start earlier. So we have completed this update by the time we want to deploy the second and more important reason for doing these things earlier is that a lot of these non-software artifacts are actually input for development.

00:13:37

And then of course, if you say a certain change requires an update of this, uh, you need to ensure that the update is being completed before you actually start development. And this is a reason why we said you, in addition to applying the queen to green mantra, we also need to think about appropriate ways to apply the shift left mantra also to the nuns of the artifacts. Very important. This does not mean that we do a huge upfront 12 months ahead planning analysis and update of these non-software artifacts. It rather means we need to do this earlier. One, two steps or sprints ahead of the software development and, um, on a rolling basis, uh, just SU sufficiently early. So the development team also have always have sufficient backlog items that are ready, uh, to develop, uh, uh, in their continuous integration chain. And if we dive even a little bit more deeper, again, the world turns out to be more complex as always.

00:14:41

Uh, we also identified fairly soon that these non-software artifacts, uh, are very, or can be quite different, uh, from the requirement when you need to do the analysis. And when you need to have the update completed, if an update is completed, uh, because these non-software artifacts are from very different sources, very different, uh, stakeholder. They are very different nature. Uh, this turns out they cannot be all treated the same. We identified four different types, which is shown symmetrically on this, uh, slide type. Number one is for example, patent application that I really require a long time to do always, uh, under the premise, if an update is required. So this should be done really early. And it also needs to be ensure that this has been completed early enough. If you're applying, for example, something like a scaled HR framework, uh, you have, for example, the planning, and in this case, we really recommend that this should be done before the planning type number two, where actually most of the nons of artifacts fall into, uh, are nons of artifacts where the update of the artifact is required before development starts, which means before this sprint starts in this picture, uh, because it is an input for development, for example, architecture documents, risk assessment, you ex, uh, specifications.

00:16:12

And so on type number three is something for example, user documentation, in order to update this non-software artifact, you really need to have the software already the change implemented because for example, you, some screenshots are required. And of course, in this case, you cannot update it before development style, but you really have to update it during the sprint and to make sure that this is being completed before you want to put this into product and type number four depicted in here is for example, open source where, uh, you are only doing the course of, uh, really detail design or potentially development. You identify with an update is required. And then also even the analysis can only be done in parallel to the development. So this is a bit getting a bit more complex, and we introduc something like a ready development, entry, quality gauge, which is a lot more than a typical definition of ready of a backlog item.

00:17:11

And it actually means that all these Korean tick marks for every change, they need to be completed. These analysis and update steps. Uh, they need to be done before actually the teams is allowed to put them into, uh, the sprint planning, uh, in order to make sure that by the end of the sprint, uh, when really from a software point of view, everything is done. Everything is fine. Everything is verified and validated. There is not invalidate. There's nothing left over on the non-software artifact, uh, point that stops us from replying this into production. What is very important if you move in this direction is that you find a way to really efficiently manage this continuous conformance. And this is really key to success. This was something we already looked into, uh, from the very beginning. Uh, and this is something like a real analysis we did quite early, where we looked at two teams, we looked at their feature and we went through all these features and through all the continuous conformance criteria, which are equally equal, uh, non-software artifacts.

00:18:18

And we looked at them and we said, for which change feature do you need an update of which non-software artifacts? And they are marked in blue in here. And you can see in here something that we already assumed, but we were quite happy when it was confirmed. That is typically only about 10 to 20% of all possible non of artifact updates that are really required. Why? Because you have a lot of changes where you don't need to update a lot of the non artifacts, or maybe for some changes, you don't need to update even none of them. And this is, of course that makes this, uh, whole continuous conformance, continuous quality assurance. So attractive that we really say, we very efficiently want to identify which nons of artifacts must be updated. And it's typically only a small set of all possible ones. And then we really want to make sure that we can identify and track, uh, uh, these tasks, uh, in the most efficient way in order to make sure that there is completed.

00:19:20

And we have a clear transparency on this at the point in time we want to deploy. And this is the second thing that was very clear to us from the very beginning that we cannot do this manual or in any Excel sheet, but we really need a tool chain. We need to do this. And ideally we should integrate it into the tool chain, which is typically already existing in an organization for the software and the software close artifacts. And by now we have done this for three different, uh, commercial of the shelf tool chains for patient team concerns, Chira and Asia DevOps. Why those three, because in those organizations where we have implemented this approach so far, uh, we have come across those application lifecycle management tool change, which were already in place, and which were, uh, taking care of the features of the backlog items of the user stories of the test cases and so on.

00:20:15

And what we did now is, uh, to integrate the continuous conformance approach. This tool chain is again, show shared medically on this slide. On the left hand side, you can see a typical requirements breakdown structure, and where you have, for example, a feature and probably already a number of user stories underneath it. And from ASO point of view, it's very clear. You have to complete all the user stories and then the features complete and you can implement it. And now we say on the same level as the user stories, we want to attach so-called continuous conformance stories in those cases where an update of a non update is required. So we could have for a specific feature, anything from zero continuous conformance stories to over a handful, to, to the whole lot, uh, really depending on the impact, this change has on this, on the different NUS of the artifacts.

00:21:12

And now what is also shown in here with some, uh, uh, screenshots is that, of course, in this tool, we have to do these two steps. The first step is analysis. Uh, do we really need, or which of these non-software artifacts need to be updated for the specific change? And then we need to up, uh, to track this, to trace this and to update it. And in all those cases, I mean, it looks differently in the different tool chains, but, uh, from the core idea, the implementation is the same. We have created templates with a redefined list of these continuous conformance criteria. You could say non-software artifacts and in the analysis step very easily just by clicking on those, you say, well, those need to be updated for a specific feature, a change and the others not. And for those that are updated, these continuous conformance stories are then created.

00:22:07

And then of course in these tool change, you can easily track the status of them. You can create dashboards and really monitor the progress. And what we really want to have in here now is that we say we have one integrated tool chain where we have the software task. We have the non-software artifacts, they are all linked on the feature or change items you want to deploy and ideal. Ideally we have two or a combined dashboard where you really can say whether the software and the non-software parts, uh, are what is still open, what is closed. And if everything is closed, if everything is clean, you are a position to really deploy this feature.

00:22:49

So coming towards the end of my presentation presentation, what I've shown you, uh, is how we really apply this screen to screen and shift left mantras. Also two nuns of the artifact in order to create a more continuous quality assurance approach. Uh, we talked about the deployment strategy about continuous conformance criteria. I also showed you about the importance of integrated this into the tool chain of creating dashboards. What I have not shown you to time constraints is that, of course, you also need to think about what, how this can be implemented into your processes, in which meetings you can do the analysis, the tracking, what are the changes and modifications to the roses responsibility and what is also very important point because even the continuous quality assurance approach to by itself is such a significant change on the way you're typically working, that you should really have a CEO driven high level vision towards this more continuous world. You need to have a suit, change management and coaching in order to drive the software across your organization. As a very high level summary, you could say, of course, whether you're an industrial regulated or whatever company, if you have products, your customer would be very happy. They would get better during the course of the lifetime every day they use them, but it needs to be made sure that it's done safely securely and with the highest in our case, industrial great quality.

00:24:23

So thank you very much for your attention. I brought two further links with you to further thought where you can dive a little bit deeper into it. The first one is a white paper I've created some time ago that not only touches the, uh, continuous quality assurance topic, but also some other topics that are relevant. If you want to apply DevOps in industrial environments, you can download it via the link or the QR code. And also what I just, uh, presented in the past 25 minutes, uh, the, from milestones to continuous quality assurance from introducing the continuous conformance concept. This is, uh, also been optical that is, uh, very soon, uh, published in the spring channel of the DevOps enterprise channel. The, as I pointed out earlier, this should not be, uh, one way, uh, exchanged. I'm very, really, very happy to discuss these topics further to address your questions, to get your feedback, uh, to please challenge me and also provide me with your experiences. What are you experience with this topic? How do you address quality in your organizations in your company? Please feel free to contact me, uh, either directly or via LinkedIn. I'm really happy to exchange any thoughts though. Again, thank you very much for your attention, and I wish you all the success. Uh, if you want apply similar approaches.