Industrial DevSecOps "Value Streams to Agile Teams"

As Agile and DevOps practices continue to challenge the status quo and improve business outcomes, large companies need to learn how to scale these practices across large, complex systems composed of hardware, firmware, and software.


The ability to iterate and deploy faster requires companies to adapt to changing needs, reduce cycle time for delivery, increase value for money, and leverage innovations. An industry-wide misconception is that this form of rapid iteration is only for software or small applications and systems.


For large cyber-physical solutions, software is only one part of the value stream leading us to consider the implication and the application of Agile DevOps principles across the entire end-to-end flow. This talk will discuss examples and lessons learned of large Government Contractors transition from Waterfall for DevSecOps both domestically and internationally for large scale systems of systems from the F-35 to Space Systems to Aegis and more. When adopting Agile and DevOps principles to developing products it is important to understand flow of value throughout the value stream and to identify the constraints within the system.


While software is key, looking at these systems holistically has brought in many other challenges. In this presentation, we will demonstrate the importance of systems thinking and value stream identification to understand flow, theory of constraints, and organizing around value.


These two government contractors will discuss not only our successes, but also where we fell short of the mark and what was learned.


The lessons we have learned and our continuing experimentation is paving the way for how companies who build safety critical system of systems can achieve similar results as the Unicorns.


Suzette and Robin share lessons learned in value stream identification to building autonomous teams in a traditional command and control culture.

SJ

Suzette Johnson

Northrop Grumman Fellow, Northrop Grumman

RY

Robin Yeman

Lockheed Martin Fellow, Lockheed Martin

Transcript

00:00:07

All right, so welcome. Um, thank you for the opportunity to be here and for joining us in the session, as we share with you industrial DevOps from value streams to agile teams, and this presentation, we're going to revisit the principles of industrial DevOps, and then describe the importance of systems thinking and value stream structures as an organization works to deliver value to its customers.

00:00:41

Okay. So my name's Robin Newman and I work for Lockheed Martin and, um, had the opportunity to deal with a number of safety, critical systems that, uh, really kind of brought me in Suzette together because similar in nature, we're, we're having to build not just agile and DevOps for small team of teams, seven plus or minus two. Most of our work has very large teams and complex system of systems.

00:01:16

Sorry, and I'm Suzette Johnson. I work for Northrop Grumman as a tactical fellow. Um, and we've been, um, on our lean agile journey for several years now. And as Robin was indicating, we've happened to notice there are a lot of similarities and the systems that we build and with the challenges that we're facing as an organization. So we want to continue to share these experiences with you and hopes that you can learn from our experiences as you go through your DevOps journey.

00:01:48

All right, so you take a look at these two systems. We've got, uh, the and the B2 bomber. Both of these are large safety, critical systems with many team of teens on them. And, um, we, we run into, um, all the complexities that you would run into on a car. Uh, so our, our approach is going to be to describe a lot of the things that we see in day-to-day life, but in the context of all set and electronic vehicle.

00:02:23

Yep. And as we go through this journey with all set, um, again, we're sharing our common experiences and how we put these into practice, um, and then using them in the scenario that I think people will be able to relate to. Uh, and as we talk about, um, a couple of different contexts, not just the system context, but also the organizational context as we expand our, um, our thinking and our experiences with implementation at scale.

00:02:57

All right. So first we're going to start with the system context, and we know this in itself is complex, right? We have large system of systems and they each got unique behaviors that we're bringing together. So as we look at it, we find that there's multiple ways of looking at quantitative information flows. The system has, you know, multiple ways that it's, it's moving through, um, there's heterogeneous elements. So we're really talking about a varied set, right? I've got firmware, I have hardware, I have software, I've got a variety of, of things that are talking to each other that are actually different in how they, um, exist, right? So it's, it's an entirely new world to kind of work across these areas. Once I put these systems together, you notice that there's an emergent behavior, right? And we have to look at all areas of the system when we're trying to find where these emergent behaviors could happen.

00:03:56

Or for example, you know, I could experience, you know, one behavior in, in a certain scenario and context and, and then, you know, move that capability out into the field and experience one that I hadn't before. Um, we have to look at interfaces right in, in the software world. We're, we're looking at API APIs and how they talk to each other. Um, and we've got software to firmware interfaces and firmware to hardware. Now, once I start looking at that, you'll notice that there's a different language, right? We're going to talk about, you know, hardware, engineers, building things like breadboards. That's a whole lot different than, um, let's say a software engineer building out hello world. So we have to learn how these different elements communicate to each other.

00:04:45

Um, so you can say there's some similarities between the systems context on the left and the cultural context on the right. And this is largely because the solutions we build and the organization of the people there, in fact, they're both systems. Um, so in the cultural context, we've also identified six related or similar areas. As we've identified on the systems contexts, we talk about the organizational structure and the organizational structure is how the organization is aligning its people and its teams to provide value. The structure is important to ensure that we're organizing around the, the delivery pipeline and to ensure decision-making is at the right levels, along with improving the flow of communication and collaboration between the teams. So for example, um, our teams aligned around features are they aligned around components, um, as they build a system, how often do their changes have a rippling effect across other teams and this rippling effect, or this rippling impact is sometimes caused by the system structure or architecture.

00:05:51

Um, but it also could be how the organization has defined their team structure. So this is an important element that requires attention to ensure the teams are able to do to, um, build new capabilities, um, and new functionality with minimal impact across the other teams. Um, if we have, uh, qualitative information flows within this organizational system. So we understand that communications may, in fact, it look different than it works structure and the way in which the teams are organized and how they need to enable value creation with fast feedback between the teams. Um, there's also the correlation between the organizational structure and the qualitative information flows. Then there is something that we're calling heterogeneous subculture. That is the impact of the different cultures of the teams, not only in their organizations, but maybe with the suppliers to we're working with who also are interfacing with us and they're part of our value stream.

00:06:54

So we bring these different groups together to build solutions, and we need to acknowledge that they likely have different ways of working. So for example, I'm a hardware team from a given supplier likely has a very different culture than another supplier or another part of the organization yet we're expecting this collaboration and alignment towards common goals to magically happen. Um, so that's an area that we want to make sure we're addressing because of its impact and how we are able to deliver value. There's shared mental models and how we think about the world in which we work and how we have shared or values and beliefs. And then importantly are the relations that ships in the language between the teams that is having a common understanding, um, for the work that we do, the levels of autonomy, um, who's making decisions. Are we able to have some level of decentralized decision-making? Is it getting at the right level? Um, what are our defined roles of engagement and all of these things are important element of the cultural context and how teams are expected to collaborate and deliver value. Um, I especially like how Ruth Milan, um, she's an expert in industry for her work on software architecture, uh, and the way that she worded it, if the architecture of the system and the architecture of the organization, or at odds, the architecture of the organization wins. So that's why we're looking at both the systems context and the cultural context.

00:08:27

Okay. All right. So here we talk about the industrial dev ops principles, and we really only have time to talk about the first one. So we're really going to focus on visualizing and organizing around the value stream. However, if you look across this slide, each one of these principles that we came up for for industrial dev ops would be impacted both by the system and by the culture. Um, but as I said, we are trying to get a lot of information in a short period of time. So we're going to talk mostly about the visualize and organize around the value stream.

00:09:09

All right. So first we begin with the system context and we're, we're talking about a fictional company, uh, all set who builds electronic cars and they are actually, um, building safety, critical systems, much like the products that we would build at at Lockheed Martin and Northrop Grumman. And as I look at the system context, there's a lot going on here. Each one of these could be considered a system of systems. Um, so I've got sensors and I have communications. I've got a variety of different products on here. I've got things like infotainment, and then I've got, you know, uh, you know, active safety. So a variety of different elements within the system are impacting the work that we need to do now for our scenario. When we talk about it further, we're going to notice that we're referring to reducing the collision avoidance system. And so by looking at that, you know, we're going to be, we're going to be impacting both hardware and software, but before we get into that, I'm going to let, uh, Suzanne tell you a little bit about how, how complex this gets with the culture.

00:10:21

That's right. Thank you. Uh, so the people that Robin's talking about, the people that, that do the work as part of these different systems that are building this autonomous vehicle, right, they are responding to the organizational context. So we've learned from experience and from our research, such as Edgar Schein's theory of organizational culture is that culture consists of both behaviors and processes, as well as the tacit values and assumptions are the belief systems of the people and how they work together on the culture will protect the organization's norms against variations or deviations. So that makes change and new ideas difficult. And, um, they may even be rejected as a result of the organization's belief system or their in, um, structure to incentivize the behaviors and performance of the people. So from a cultural context, we start to recognize that there are multiple aspects coming into play a lot with the technical considerations of industrial DevOps transformation. So as the organization starts to embrace a new way of working, it began to experience the shift in culture, a shift in mindset, which needs to be part of their transformation plan as well. So the activity is to support the culture change are planned, just like we plan the system capabilities. Um, and we create roadmaps for that change. That we'll talk about a little bit later.

00:11:51

So here we're, we're taking look at, um, the past papers, right? We began with industrial DevOps in 2018 and came up with, uh, principals. And then we had, um, additional folks that were added to our team. And we went through and we created a fictional company, all set, and we applied those principles on a real life system. Now, the interesting thing that we've run into in 2020 is that story has taken a turn. As we looked at it, we're like, Hey, we did this great thing. Technically, why did we have a problem? Oh, we forgot to bring culture into play. So the key that you're going to get from the 2020 paper is the discussion about how culture impacted this journey. We focused heavily on the system itself, um, but not necessarily looking at the system of systems, which included all of the people I'm going to begin with, um, the, the business goal, right?

00:12:54

To improve collision avoidance, the great, uh, concept. And so we want to, uh, decrease the closing distance by 50% quite a goal. And in order to do this, we created two epics. Uh, the first one really was to enhance obstacle detection and it's for the cars that are currently in the fleet. We're making the assumption that we have a set of cars in the fleet and that we have another, um, set of cars that are waiting to be built. And so that set of cars in the fleet, we're primarily just updating software, right? So, so not as big of a change as, as you know, the second one is, which is where we're going to update the camera. Now the camera takes into account a firmware and the hardware and the software. And you're going to see that while this looked pretty doable from a system perspective, we ran into some issues. Um, so, uh, so does that tell you about the cultural impact?

00:13:55

Yeah. So in our scenario also, they were actually able to finish the development of the epics. Um, but yet due to some of the cultural impacts, we weren't actually allowed to ship. And that's because due to these pressure and, um, the constraints that they, and the need to deliver quickly and the pressures that were put on them, um, they found that they were actually shortcuts taken in terms of quality. And this was identified during their demonstrations. Um, so while the functionality was created, they couldn't deliver. And the root cause or the underlying challenge and our scenario was that it wasn't the technical competency at all, but it was rather the culture, specifically the organization's management culture. And that was that due to pressures to deliver, they deviated from their, their sort of core lean values of building quality in, and they started taking these, these shortcuts.

00:14:50

Um, management's along with how people were incentivized to meet schedule demands at all costs. Um, the team members started also feeling like they couldn't speak up, that they weren't empowered, um, that they were feeling like they were in an environment that was not psychologically safe. So that way that resulted in the teams not speaking up as they were recognizing that they were having these concerns, because they were, um, concerned about the negative impact either on their performance reviews or how they might be viewed within the organization. So while they had the technical competencies, the organizational and cultural impacts were actually keeping them from delivering their high quality products that they've done in the past.

00:15:35

Thank goodness that this is only fictional and couldn't really happen.

00:15:43

All right. So I'm going to look at the all set value stream identification, as we discussed. We're going to take into account that, that first piece, which is organizing around the value stream. And as you look at it, you can see that this actually decomposes into multiple value stream Lutz. Uh, it starts getting even more complex, right? So I've got collision avoidance obstacle detection and traffic indicators within a variety of product lines. Some of these product lines are actually being delivered by suppliers. Um, so, so there's definitely going to be variances in, in the products. Um, and then they can all be deployed to a variety of vehicles, right? So trucks, performance vehicles, or hybrids. So coming back to those original items that we talked about for the system, right, I've got that complex system of systems where all of these different elements are talking to each other, the information flow from obstacle detection to active safety, um, you know, could have an issue depending on which context, the actual vehicles in, uh, these heterogeneous elements.

00:16:52

So when we say heterogeneous, the different, uh, hardware, firmware, software elements, the electronics, right? I'm, I'm not doing it justice because there's so many sub pieces. When we start to break this apart, um, emergent behavior is always an issue. And, and the reason is right. It's, it's just, you don't know what you don't know. And this happens all the time, uh, really looking at those interfaces. And one of the keys we've found is to make sure that we're building around stable interfaces, right, is impacting this, these value streamlets. And again, coming down to the different language, the different nomenclature, um, if I've got a supplier and I have, you know, a prime or I've got multiple suppliers, it's likely that they have products that are the same, but actually name something different or are they have a slightly term in their database. Right. And I've got to connect these items together. So there's a lot of complexity going on just with the system. So what happens when I add people,

00:18:05

'cause people do all the work. Um, so the complexity of the systems context is actually amplified and the cultural context, um, all those value streamlets are comprised of people. Um, and the teams working for all set and they started facing several challenges. Um, and these are similar challenges that we've actually seen and through our experiences and other domains where the world of the software development teams, um, in some cases has been very separate or very isolated from the hardware teams. And now we're asking these different cultures, these different environments to come together on a regular cadence to organize their work together, to plan together, to do demonstrations together, um, with having to adopt new tools and new processes. And this is creating a great deal of strain on the organizational system. So some challenges that might be experienced include things like different parts of the organization saying hardware versus software have their own subcultures not aligned with the dev ops mindset.

00:19:06

So this means teams may have completely different ways of working, um, making alignment of the integration, difficult making reporting, nearly impossible. Um, this means teams may not be able to look across their value stream and collect any lead in cycle time metrics. They're not able to visualize the flow across the system. It becomes very disjointed. So the, for the teams, in which more frequent collaborations required, this becomes a growing and exponential problem. As you scale, also agile hardware firmware and software engineers. They do not have the same mental model, maybe around collaboration and how they do work. So for example, hardware teams within our scenario of all set, or maybe a different domain space, um, they have a different way of working. They might be struggling from a hardware perspective and this kind of, um, environment where frequent feedback is desired. So that means they have different way of working.

00:20:05

They have different technologies they're dealing with. They might be making use of new tools, digital twins, increased modeling, and this is all impacting their day to day work experience. So again, a lot of pressure and a lot of change in their, in their experiences and how they do work. So how do we address some of these challenges? Well, one way is we create an always focus on the small cross functional persistent teams, um, that share a common set of practices. Um, we continue to support self-managing teams explicitly agree on the interaction, um, modes and how they want to collaborate and setting expectations, um, to help shape their set of behaviors, that mental model of how we expect to collaborate, to build a solution. We create not only a system architecture, but an organizational architecture, that's designed to reduce some of the handoffs while improving the interfaces.

00:21:02

And that is not just the system interfaces, but the interfaces of the teams, the collaboration points between the multiple development teams using clear, defined set of practices. Um, people must also see their leadership modeling these behaviors, um, and actively engaged in the process because leaders model the way, um, and another area to address is technical competency resign because we have this new way of working. We have all these new technologies, um, all these new automation tools. So it's up to the leadership team to ensure that the people have the skills that they need, um, and the environments and the tools and the resources and the opportunities to gain these new skills, um, and making sure that's happening for them. So there are some definite challenges that come into play, um, through the organizational context and the culture of change. Um, and we continue to look for ways to address and alleviate some of the strain as we define a new way of working together.

00:22:07

All right. So one of the things that we came across, then we looked at what was actually, Hey, this the system roadmap, right? Um, we've had this all along. We think this is a great thing. Um, and it, it allows us to look at the system and be able to make incremental changes on a regular cadence, right? So we're coming back to another one of those principles and the four kind of key areas that we looked at were the system, uh, prototyping, refactoring, and innovation, right? So from, from our perspective, we, we said, Hey, if we did quarterly review of our plan and then regularly updated that, and then actually brought it together with, you know, both sides of the coin, the system and the culture that we could incrementally move the system and the culture to where we want to be. So here, what we're looking at in quarter one for all set is to kind of validate what's the as-is state, right?

00:23:08

And what are my, my business objectives. And, and you heard earlier, we want to decrease that, that closing speed, um, you know, prototyping and really looking at the system telemetry, perhaps, you know, all set is starting to see things like, uh, you know, slow development speeds or, or is taking a little bit longer than expected. And a lot of times in both sides of those, we're, we're looking at a technical debt problem, right. Areas of the system where we've made changes, but maybe we haven't cleaned up everything. So each one of different areas, we can actually intentionally try to move the system and make it better and better each time. And we can actually add that together with what, so that's going to explain.

00:24:04

Okay. So as part of the industrial DevOps journey, we've come to understand the importance of organizational culture, obviously. Um, so in our scenario, we demonstrate through all the importance of setting up, um, a guiding coalition, excuse me, or a leadership team that addresses and starts thinking about the needed success patterns, um, principles and mindset for this new way of working. And what does that mean? Um, and how do they want to make this change at scale in order to achieve their desired and hope for results? So just like they did on the, um, the technical side, they also create a roadmap for change. Um, and our example offsets a guiding coalition, created four areas of focus, um, mindset, validation, support structure of the organization, technical competency and role modeling behaviors starting with leadership. So they started laying out their, their roadmap for change. This is where Matt will evolve over time, just like a technical roadmap would evolve over time.

00:25:13

Um, they focus on building awareness, um, sharing case studies that demonstrate, uh, and help visualize what successful change looks like. And as they begin to see positive results within their organization, they also capture the successes and share them across the organization because we know success will build on more success. They ensured alignment with their promotion system to include team-based awards and incentives. Um, the importance of building a learning culture with regular opportunities to grow was recognized as a key component to the successful adoption of industrial DevOps. Um, it, they recognize it requires new skills and competencies, and most importantly, they learned the importance for leadership to lead and model the desired behaviors, their leadership made commitments to understand and model the principles to build environments of psychological safety and decentralized decision-making.

00:26:18

So what are we looking for? What's our next steps? Well, we'd really like to have other examples of value streams for large cyber-physical sins lotions, right? So we have a bunch of Northrop, a bunch of Lockheed, but if you have others, we would like to continue to research this and get better. Um, we would love to hear how your organization address leadership and any kind of cultural challenges you've had for large transformations, right? So it was Peter Drucker said, you know, culture eats strategy for breakfast. We know that. So we want to, we want to hear your stories, any information, uh, bridging the gap between manufacturing and development, um, across different phases of the life cycle, uh, for, for our products, because I think this is really going to take us to, to the next step, um, potentially next year's paper or, or even more,

00:27:20

Uh, and finally we would want to make sure we are recognizing our other contributors and co-authors, um, while Robin and I often present together, um, on our papers and what we've, um, coauthored, uh, we definitely want to make sure we give recognition to these other individuals who have put in a lot of time, a lot of effort to help us author these papers and to review and provide feedback. Um, this is definitely a, um, learning experience and contributions from quite a few people, and we just appreciate their, their time and energy and making this, um, happen. So thank you.

00:28:04

So in summary, leveraging the power of industrial dev ops for large complex systems, um, is an industry step change, right? It's going to make a difference and the companies that solution, the problem first are going to, you know, experience things like transparency, reduce cycle time, right? We're going to be able to get capabilities out the door, increase that value for money and innovate faster. And, and just a, a quick side note, uh, you know, Suzanne and I both talked about the system and the culture, and there's kind of an interesting story behind that. So currently I'm working on my PhD and I'm getting my systems engineering degree, and I'm focused on the system aspect of, of agile and dev ops. Um, but interestingly enough, uh, that you want to tell them about yours.

00:28:57

Yeah, so I, I graduated about 10 years ago now with my doctorate and my dissertation actually focused more on the leadership and the organizational aspects, um, and looking at the behaviors and practices within the organization and the impact on outcomes in both agile and waterfall environment. So it's kind of interesting the turn that our, um, papers have taken over the years and just kind of aligning with our interest in our, at our dissertation studies as well. So it's, um, a really neat opportunity to be able to leverage that research and, um, the effort put into that.

00:29:41

Alright, well, thank you very much.