00:00:02

I'm Suzanne Johnson from Arthur Grumman and Robyn from Lockheed. And you may recognize, um, our, our companies, if not, we are in the Aero defense industry and we build very complex systems for department of defense and we are competitive mates. And what that means is sometimes we compete. Um, and sometimes we are our partners. So today we're coming as partners with a common message and a common set of experiences that we want to share with you.

00:00:30

Absolutely. Um, so this is, this is at night at maybe interrupts, but, um, we, we tend to partner pretty frequently, um, when we're speaking, because most of the projects we work on are so similar. Um, so take us to the, the reason we're here today. Um, we're going to present, um, industrial DevOps. Uh, now we couldn't really do something that we build actually. So we had to do something that was similar in nature. So we partnered with a larger group in order to build something that would be similar, but we'll try to give you some examples from our environment as well. Um, we want to talk about the discussion on how we build products, not just the software. So a lot of the folks that we work with have this perception that we're just going to be talking about software and the everything that we built from, uh, the AFT 35 to the B2 bomber, um, and everything in between requires software, hardware, and firmware. So we're really passionate about making sure that we can look at the entire value stream and how we can optimize that delivery. All right.

00:01:49

Okay. Right. So what we did is we took a scenario, something that we could actually talk about and actually get through our external release process. Um, so we chose this autonomous vehicle, uh, this, this, uh, car, and it has a lot of similarities. Of course it has different capabilities on which are the things that we can't talk about, but we can in this scenario. Um, so we have an autonomous vehicle and we're looking at just at the software firmware and hardware as part of this scenario. And what does it mean to apply DevOps principles into this scenario? What does it look like? How can we start evolving this beyond software? Because one of the things we recognized is if we built all our efficiencies, only in software, there were other parts of the value stream that were being ignored. So we want to take a more holistic or systems perspective.

00:02:40

Awesome. So we're going to talk to you. Um, and we, we did, we took this from, or it's, it's very similar in nature from, from scaled agile, cause we've been working a lot with those folks as well. Um, hence you know, the scaling perp. So, um, we're gonna look at collision avoidance, um, and increase right obstacle detection systems, uh, by, by 50%, which would be something similar that we would do, let's say when we're building, you know, automated drones or autonomous drones. So, uh, the enhancement of our collision avoidance capability is going to be handled through three things, sensor, firmware, um, uh, control system with software and a user interface. Uh, this would be very similar nature to how we would break down an existing product. So we discussed two key epics, right? Large cases, uh, in order to demonstrate that actually agile and dev ops works really well for the entire value stream, not just for the software portion. So we're looking at both the camera and updating the camera in future vehicles. And Sue's, that's going to walk you through the first set of practices, all of them actually walk them through.

00:03:58

Um, and what we'll share with you at the end or where we've been publishing this. So the first thing that we did is we looked at what are the principles, what are, what principles and this type of scenario would we want to implement or to align with in this situation? So we came up with eight of those and what we're going to do in the next set of slides, the next set of eight slides, we'll walk through each of these principles and then how it's applied against the scenario. Okay. So the first one being, looking across the whole value stream, again, this is where it became really important because what we noticed as well as we were applying, um, agile and DevOps principles and practices and software, there was the rest of the value stream that was being ignored. So we wanted to take this perspective of let's look at let's look holistically.

00:04:47

So with any of our transitions, one of the first things that you probably look at, even in your organization is what are the value streams and how do we want to get down into the, the, um, what we're calling the stream lit level in terms of delivering and defining these larger epics? So in this situation, we looked at the vehicle as being the entire value stream of them. We looked at the safety and security as being a sub component and then zeroing in, on collision avoidance. So that's what we're calling our value stream lip. So with collision avoidance, we have two areas that we're focusing on, again, the software and the software updates to vehicles that are already in production. So we want to improve our collision avoidance system. And there's a set of vehicles that are already in production and we'll make those changes and we can actually feel those, but then we have another situation of that.

00:05:40

We want to make those improvements, but we want to make those improvements to the new set of vehicles that are going to be released. They're still in development. So looking at a value stream perspective, we're going to look at the software, but also the firmware and the hardware around that situation and that we're wanting to improve. So if we go to the next slide, so what do we do next? Well, we think about, um, the, the value stream and the stream lit, we think about collision avoidance. And then we always start with the vision, what is the vision that we're after? And then what do we want to do? And we start taking a year, look a year long perspective, um, again, both on the side of cars that are in production and with the vehicles about to go into production. And from that perspective, we want to enhance and improve the camera.

00:06:33

So with vehicles that are already in production, we're just going to make software changes that improve the collision avoidance and reaction into the system. But for the ones that are going to be released, we're actually going to do hardware, significant changes. So we're going to take a year long perspective and think about both of those types of ethics and what do we need to do, um, both yearly. And then how do we start taking these quarterly views? And the quarterly view is really important because we want alignment, right? We want to make sure the software teams are also talking with firmware, also talking with hardware as part of this collaborative effort. And then of course, the more familiar that we get into, or the iteration plans and the daily plans. Um, the other thing around this is because hardware is unique. It's not, of course, exactly like software. There's some challenges around that. Some of that's the longer lead items, which is also why we need to take a year-long perspective. And with those longer lead items, we have to get those scheduled. And then we also have to have targeted milestones of when we might be receiving equipment or how we be working with our suppliers. So that's why these multiple horizons of planning become really, um, important and significant for these larger solutions that we're developing.

00:07:50

So once we're taking that perspective, then what we want to do is as we're thinking about each epic and think of an epic is something that's going to take, you know, six months to a year to create a feature would be something we would develop within a quarter. And then stories are iteration to week level type of, um, backlog items. So what we want to do, though, with each of these starting at the epic level is thinking about what would we actually demonstrate? So this, um, what you're seeing here is on the software side, it is looking at being able to, with that epic being able to drive the vehicle through multiple scenarios, right, and come to actually be able to test it, demonstrate and show that we've hit those, um, outcomes that we were expecting in terms of improvements on the hardware side, it might be looking at some prototypes, we'd have to be testing in some prototypes and getting regular visibility of progress against the prototypes and improvements that we're making on the hardware side. Um, there might also on the hardware side, be some research items that we have to do, um, our student designs as well.

00:08:53

Okay.

00:08:56

Yeah. And then most importantly is architecture. Um, so we understand that the software level and at the lowest level architecture emerges, we get that. Um, but also when you're building really large systems, what we want to do is think about architecturing for modularity. And we have to think about that, um, for the bigger system, a little bit ahead of development. So here's our architecture. And with this scenario we focused on primarily on not just the, um, improvements in the sensor management and the light detection, um, also the reactions, um, and this how it integrates both with the braking system. This is our management and overall vehicle control. So as we're going through and we're planning across those multiple horizons, and we're looking at the different backlog items, we can also then align it with our architecture and seeing where the impacts are in the architecture. So what does this start to mean? That means as the teams are planning together, we have to start defining the dependencies in the architecture where they exist, uh, because, uh, especially on the epic too, where there's hardware components, we may actually start talking about modeling and modeling and simulation because some of the hardware might not be ready. So that would be part of how we're looking at this whole picture from our architecture to our modeling and the modeling simulation approach that we're taking.

00:10:17

Okay. So I'm going to take the next couple, um, again, uh, many of the systems we built very similar. So in this, in this case, uh, iteration and batch size is still critical. How many of you guys have had, you know, hardware folks on your team or have had build hardware within your, your programs? How many of them come back and say, I can't actually get anything done in, in a week or two weeks. Yeah. It's not possible. And, and the reasons they tend to come back with that, I think is that we're really looking at, uh, things like how to break down our work. I think that that's where the struggle is. Um, so Suzanne and I are both very passionate about systems engineering and how to actually decompose your work, because it enables this ability to do what we're talking about here, which is iterating and reducing batch size.

00:11:11

So the first epic that we're talking about is that software epic in this case. And so I can start delivering some upgraded capabilities or evaluate those, right. Simply by injecting them into the system. Um, and we will be using, you know, for, and we are for, for many of our systems, things like a digital twin. So digital twins do emulation simulation, right? So I can take a cyber physical system and recreate it in a digital environment by recreating that cyber-physical system in a digital environment, I can start buying down risk in all different kinds of areas in much shorter cycles. So we're going to do that with our software to make sure that we don't impact any of our, uh, you know, our current users, which would be critical. Um, and then we're going to iterate on our new camera. Now, just like Susan was talking about, we can do some 3d modeling.

00:12:07

We can leverage our digital twin again, to determine the right camera, to buy the right way that it's going to interact with our vehicle. Um, and we can do small demonstrations. This is, uh, again, something that's been around for years. I keep telling the hardware guys who tell me they can't get anything done in, in less than two weeks. I said, actually, software did what software does, which is copy and paste this from, from you guys in the first place. Because if you're looking at it, it looks a whole lot like lean manufacturing, right? So that's what we're looking at for our car when we decompose that down to, to get you something out and using that objective evidence that Suzanne talked about earlier, all right, next thing we want to do very similar is apply cadence and synchronization the differences. Not all of them have to be on the exact same, right?

00:13:06

We want them, the integration points to line up, but if it, but if I do have a long lead item, when purchasing, let's say custom hardware, or, you know, let's say for something that we would build, maybe I've got to look at, you know, uh, satellite panels and I need to understand, you know, how they impact with, you know, radiation. I have a long lead item. So while I want to make sure that my integration points and my teams are all working on the same cadence, you could see here that I might give myself a little longer with that hardware, right. And firmware, I might say, well, those FPG AEs or Asics. I might give myself a little bit longer than I would with software. And when I get down to software, right, I may give myself more time, right. A longer time on, let's say, real time embedded versus, um, you know, software that doesn't actually have any of those timing constraints. So this is how all of those teams can actually be on the exact same value stream and be delivering objective evidence against the capabilities. Now save, usually refers to this as, um, the system sprinting, as opposed to the team's sprinting. And that's, we just built upon that and said, Hey, we want our car to sprint. We want to get, you know, objective evidence for that capability out soonest.

00:14:33

Now, the next thing that we talked about in our epic is, is really looking at, um, how quickly you can integrate, right? So I've got teams that will never do continuous integration. I know not everybody loves the term continuous ish, but they'll never do continuous integration at 16 would be a good example of that. It's built on an ADA baseline and believe it or not, there's not a lot of automated test tools out there that support ADA. So while originally, um, they were integrating every three or four months, we got them down to integrating every couple of days, that's as close as they're ever going to get. But quite honestly, that's an amazing feat, right? So that's why we talked about it with our vehicle to we're going to leverage the digital twin and we'll allow our teams to, you know, be able to visualize updates to the code will allow them to, you know, do all of the sensor management, things of that nature.

00:15:32

But maybe for some of the real time embedded, they're not integrating every day or maybe they are. It depends on, on the, um, environment and tools that we have available. Um, the new camera, that's going to be a much longer integration item. So as we're looking at that new camera device, we want to understand how it's going to fit with the car. Um, we need to understand how it works with our current API APIs, um, any kind of, you know, uh, mechanical issues, uh, that could possibly happen when we attach it to the car and all of that's going to have to be prototyped. So, so that lead time, right? They're not going to integrate every day, but maybe they could integrate with prototypes once a week or maybe they could integrate every few weeks or maybe they can integrate quarterly. The bottom line is the faster I can integrate whatever I'm building, the less risk I have. So I'm not trying to, you know, get away from getting that fast feedback. And that's kind of the thing that we've run into with a lot of our larger systems. So that's why we called it continue. Ish. I would tell you if you come up with a better term, let me know. Cause I have heard that bashed a little bit here and there.

00:16:51

Test driven. So be test driven, begin with the end in mind, this also frequently comes up with our hardware engineers. They're like, we can't, we can't start with the test. I'm like, well, why, why is that? If I don't know how I can prove what I'm building, then, how am I building it? Um, and what happens, you probably have seen this with some of your teams is they keep building until they've got something done and it could be that it's exactly what they need, but usually there's more in there, right? It's like Prego it's in there. Um, so it's not quite as clean. Um, I probably built some extra things that I may not have needed. So with our approach, we're going to have them use acceptance test driven development. So we're going to start looking at how I'm going to validate and test this work before I even begin updating my software or my hardware.

00:17:45

Right. And using things like MBSE is going to allow me to test that model out. I'm looking at how I'm going to do validation, even real validation. Right. So in, in person, I want to make sure that I actually have two cars and I can, I can determine what that, that collision breaking distance looks like. So beginning with how I'm going to validate that versus trying it afterwards, or just testing it afterwards is going to make a better product. And we didn't invent this actually, Joe Justice showed us exactly how to do that, um, with building a vehicle, if you guys are familiar with that. All right.

00:18:27

Great. Yeah. So, um, a couple of items, first of all right, the importance is going back to the why. Remember in one of the earlier slides, we said, why are we doing this? And some of the points we touched on was the need to burn down risk and looking at flow and feedback across the whole value stream, not just in software, we have a couple more things that we want to talk about. One is our contributors. Um, and Harry is actually here with us. Um, um, Josh is somewhere around. I'm not sure of the others. Haven't crossed paths with them yet, um, at the conference, but, um, we appreciate their input and helping to write the paper. So I think if you go to the next slide,

00:19:05

We got a couple more, Steve Maynard is actually here. And so is Dean Leffingwell. So we have, um, you know, what, a lot a team with a lot of passion around, uh, dev ops. And we got some really good input from all these guys. All right.

00:19:24

Yeah. So help that. We're looking for. Um, as I mentioned at the beginning, we had those eight principles. So year one, what we did the group of us is to figure out what were the eight principles that we wanted to talk about. And we defined what those eight principles are and why they were important as part of what we're calling industrial dev ops or dev ops for large cyber-physical solutions. This past year, we wrote it, we did another article, which was taking those eight principles and then putting them into the scenario with the autonomous car. So we could start looking at how would you actually implement those principles? So next year we want to write the, the next paper. And here's some things that we are thinking about, and we want you to help us figure out what that next paper should be. So based on what you're hearing it thinking about, well, you know, that's really great, but it really helpful if, and then you help us figure out what that is. Is it more metrics? Is it more systems engineering? Is it how the, how we actually trained and would get the people up to speed and the skills and the technologies needed to do this? Um,

00:20:33

Yeah. And, and any kind of, what are all the aspects of this value stream, the content. So things that we're being impacted with right now, um, just, just as a side note is, uh, the government's making massive changes in the acquisition policy. So we've been working closely with them as they do that. Why? Because we need to get capabilities out to the warfighter quicker. So when I look at this, I want useful, I want to build useful content that will enable our programs, who are building capabilities for the war fighter to be able to get something out there quicker. And things that I think would be incredibly helpful is okay. So how do I decompose a system where I can show vertical slices? That's that's one way. Um, so we want to look at maybe there's some additional assess systems engineering. We need that content.

00:21:24

Um, another, uh, thought is how do I do test automation? Are there other practices? So is that something that would be valuable and useful, um, to the greater community? Can we build some of that? Um, could we look at improving, you know, H how we get those cross-functional teams or, or do they need to be completely cross-functional? So that might be, um, a place where we could, uh, expound upon this, or should we try to put more systems through the paper? Right. So the first year, as Suzette said, we created the principals kind of walk through it second year, we actually took that and applied it to a vehicle. Maybe, maybe we need to look at applying it to something else. I keep, I keep bugging her about this awesome, uh, cube set that's out on get hub that I'm like, oh, we could, we could build the cubes and we could 3d print it, and we could, uh, update the real-time with Arduino.

00:22:20

I'm I'm all in. So do we want to push something else through the system? And the reason I'm excited about that is because I actually worked for LM space, which so it's, it's something that, um, I think would be near and dear to a lot of my team's, uh, hearts. Um, so I think there's a bunch. And how do we want them to give us that data? Like either email or, um, you know, do we want to cry, for example, the LinkedIn, LinkedIn, um, also if there's anybody who does have passion around helping to build this content, you want to join our team, we're always looking for help. That would be awesome too.

00:22:56

All right. So here are the papers and they're actually here, right? They're printed here. These are the two ones we did, um, defining the principles and applying them. Um, and I think that, uh, you can access them both on the website as well as get a published copy here, but I'm not entirely sure about the published copy. So we'd probably have to go look at that. These things I believe are going to add value to the ways we're working, right. When gene talks about the way, uh, one of the best things that we've seen is the huge transition in how people are delivering software. Like the unicorns. The, the thing that I think would really, really help is taking that way of building our digital products and adding that to how I can do physical capabilities, real live physical things that you can see and touch. Right. So how do we imply industrial dev ops to building ships? Or how do we apply industrial DevOps to building radars? How can I get capabilities out the door quicker? Uh, no matter what domain you're in, whether it's in the government domain or the commercial domain, because we think both the life cycle is getting shorter all the time.

00:24:17

All right. With that. Um, we actually have a few minutes for questions. We probably went a little fast, so, so does anybody have any questions about this? Any interest? Yes.

00:24:36

So however you get the secure,

00:24:43

That's a good question. Um, might be another piece for the paper. How do you get security aligned with these principles? So while we're giving faster capabilities, how do I make sure that I'm building that security both into my software and the hardware? Um, and you're right. That is also something that we've been wrestling with because a lot of folks will say we're building safety, critical systems. How do I get a safety critical system out the door? Now the answer to that question is we haven't resolved it yet, but it does seem like a very good option for, for the next generation of this paper as well.

00:25:19

Yep. Yeah. I was just gonna say in general, right. We build the security and the security requirements in, but when it comes to external entities that need to work with us, then that's a collaboration opportunity.

00:25:30

Yeah. There's going to always be, you know, flight qualification. There's always going to be, um, external, external folks that are accrediting our work. So that makes sense. And in some cases, yeah,

00:25:41

Is they have engaged depending on what the system is and who the, um, the entity is that is securing and reviewing the software. Um, in some cases they're more engaged and they understand what we're trying to do. Um, so that helps in some cases they're like, like I said, still opportunities.

00:26:02

So within Lockheed, um, I can tell you a couple of the systems that we've actually been applying some of this to. Um, so fleet ballistic missile would be a good example. That's a, you know, across, across the U S so we have them in Sunnyvale, California, as well as, um, down in Florida. And they have been doing a lot of this for their hardware, um, Aegis, which is one of our biggest, um, agile teams. There's about 190 teams that are building that combat system. Uh, we also have at 35 just portions of it, but you're going to see over the years that, uh, 35 is going to transition much more into this type of capability at 22 actually has already done a few presentations about how they integrated the software, the hardware and the firmware cycles. Um, they're not perfect yet, but you can see that they're also building out content. So a lot of this work is very relevant to, to what we've been doing. Yep. All right. I think we're good. I think so. Okay. Thanks guys.