Las Vegas 2018

DoD Science Board Report

Robin Yeman works for Lockheed Martin (LM) Information Systems and Global Solution in Northern Virginia as a Lockheed Martin Fellow.


She has over 20 years of experience in software and IT, across multiple business areas building everything from Satellites to Submarines.


She has been actively supporting and leading DevOps programs at Scale both domestically and internationally for the last 16 years with multiple certifications including Scaled Agile Program Consultant, Certified Scrum Practitioner, CSM, CSPO, PSM, PMP, PMI-ACP, INCOSE Certified Systems Engineer, and ITIL Practitioner.


She actively coaches and trains teams through in person coaching, Agile workshops, virtual training classes. She leads the Lockheed Martin’s Agile Community of Practice and Center of Excellence, speaks at multiple conference engagements each year.


Robin received her Master’s Degree in Software Engineering from Rensselaer Polytechnic Institute.

RY

Robin Yeman

LM Fellow, Lockheed Martin

Transcript

00:00:05

Right now we're still headquartered in Bethesda, Maryland. That hasn't changed. Um, we've got just over a hundred thousand employees, um, building pretty much everything from submarines to satellites to intelligent systems. It, it doesn't matter. Um, we've got five lines of business. Each one in their own right could probably be a, a company. So we've got aeronautics, um, uh, missiles and fire control, what we call RMS, which is Rotary Mission Systems. It's really the integration of the Sikorsky helicopter and our surface ship stuff. I don't totally get the, uh, the link there, but so what we have, um, space, and then we've got internal EBS.

00:00:48

So we began our agile journey in about 2002. And it's not because we decided that we wanted to be on the bleeding edge, because that's not usually our, our jam, but the intelligence community started requiring agile pretty early. So we've gotten, uh, RFPs a request for proposal. Um, we began our DevOps journey in 2012, and I, I remember the day that I came back and, uh, I presented at our, our systems engineering and architecture internal conference, and I said, we need to go to DevOps. And they were like, what is that? And why would we need to do that, Robin? So, lo and behold, you know, persistence over the years, uh, made able to get them to move. Um, so this year, this year, a couple things I'm pretty proud of. Well, the one thing I'm proud of that we've been able to accomplish is building a software dojo.

00:01:40

So in 2015, target did their presentation about their software dojo, and the first thing I said is I want one of those. Um, and it only took me three, four more years to get it right. But on October 24th, which is tomorrow, it will be officially done. So I'm pretty excited about that. Um, and then what I was gonna tell you is we've had a lot of impact this year, and the reason for the impact really came about from something we call the the DOD Science Board report. So I'm gonna tell you guys a little bit about it, but it's really why the, the DOD, the federal government, the intelligence communities, all of those things are going to be moving to DevOps in a big way fast.

00:02:24

All right, so just a little bit about the landscape here. You can see we've got, you know, our defense strategy and, you know, the key members in that defense strategy and how they, um, they, they put together what we're gonna build, what's, what is, what does the nation need. Um, they've hired somebody called Jeff Bowling. He came outta Carnegie Mellon or SEI, and, um, he's going to lead software for the DOD. Now, the reason is, is because one of the highest risk areas, or the highest risk area for the DOD and cost overruns happens to live in software, right? So this year they were like, we have got to get a handle on it. So here is a, is a summary again, of the defense strategy and some things that were said during that timeframe. I'm not going to read it to you 'cause I think you guys can, can read it, but I do highly recommend that you go out here and get these reports if you have any interest in kind of, um, development or government contracting or anything like that. Again, some other hugely important people in here. You've got Ellen and then Jeff Bowling. He's the one that I told you has just come on board.

00:03:42

And one of the key things that they're all saying is software is, you know, the thread through our prog programs. It's the important piece. Um, and a lot of things that they mentioned I thought were just brilliant because, you know, the commercial industry got it a long time ago. So agreed. We're at the inflection point. So just like mc Kirsten's, uh, presentation last night, he was kind of talking about, you know, where we were. We are at this inflection point, and the government knows it too.

00:04:16

There's a DOD software task force, right? So Mitre helped this, and this is the presentation that they put out, which is the DOD Science Board report, but I'm more interested in telling you what's in it, right? So they, they gave about seven recommendations that are really gonna impact how we deliver, uh, products, services, capabilities for the war fighter, how we get things faster. One of the also interesting things, if you guys do pick up this report, is they actually put a picture of the F 35 in there, and they pointed to it with an arrow to say this would be a good one to transition. So I think they thought we didn't get the hint before, and they thought if they put it in pictures, we would be all over it. <laugh>. I'm just saying, all right, so what did they say? What's, what is this magic?

00:05:01

And some of these, actually, many of these I am sure that you guys already knew were important, all right? You're already doing, um, one, they interviewed a variety of companies in Silicon Valley, and one of the things that they kind of found out over and over again is programs, projects, value streams, whatever that are successful, have a software factory, right? So people, process and tools to be able to deliver business outcomes. Cool. Continuous iterative development, right? Before we were doing, you know, our traditional waterfall, do all the requirements here, they're saying, you know what? They're not calling it agile, but they want continuous iterative development so that they've got transparency. Terrific. My favorite risk reduction for new metrics, we'll go into detail in each of these in just a few minutes, but one of the things after this one, I, I asked my boss if I could get a T-shirt that had the word lock in it in a circle and an X in it, because we've been measuring Slack for as long as I can remember.

00:06:02

I've been telling 'em it's a bad idea for as long as I can remember, and I'm really excited that they were like, yeah, wrong metric, great vanity metric. That's about it. Um, they made the point to say it's not just new programs. They want to transition. They said, any program, current legacy in development, production or sustainment, they want us to transition, all right? Investing in our workforce. Uh, right now, there is an actual shortage of software engineers, at least in, in our space, and I, I believe for, for every space. But it's really noticeable in the government space because a lot of them need to be cleared as well. So finding a, a top secret cleared software engineer with a background that has, you know, DevOps and agile skills is, um, is like finding a unicorn. Like we are just very short. Um, they also mentioned that software was immortal.

00:06:59

Again, I work at Lockheed, and I can tell you for the last 25 years I've been hearing it's just software, right? 'cause we really believe in the hardware. So afterwards, I had to send a note out to all of our leadership to say, actually, you know what? I heard that hardware is nothing more than a mechanism to deliver software, making friends and influencing people. All right? And, um, and then we said, we should be moving to IV and V use machine learning for IV and V. So IV and V for us is integration, verification and validation.

00:07:35

So how are we gonna do this? Oh, the other thing I didn't tell you is this report came out in February of 2018. In August of 2018. This report was actually, well, the report itself wasn't, but there was actually legislation passed that said all government, uh, folks that are working with government dollars have to transition within the next 18 months to leverage the recommendations in the DOD science board report, or explain why not, which will be quite a challenge. So, um, apparently we didn't move fast enough, so they passed legislation. All right, so what are we doing? Well, been working on the software factory for a long time, and actually we have multiple, and they're in different business areas. Um, and so one of the things that we're working on is trying to, to standardize that, but really standardize it at the data layer. So, as much as everybody wants to say, we're going to use the same tools across the entire platform, by the time I switch any set of groups to a specific tool, I guarantee that the next tool will be out, right? So I really want to make sure that we leverage standardizing at the data layer and integrating those tools, um, because I think we're, we're at a place where we have to assume that by next year there'll be something amazing that I haven't even heard of or thought of yet.

00:09:05

And also, we've been able to take that software factory, like I said, and put it in the software Dojo. So maybe next year I'll be able to tell you about results. I'm really hoping. So

00:09:15

Here's the example. This is just one example, one instantiation, there's many. Um, I did put tools in here. I wouldn't wanna get into a tools war because actually I'm, I'm not that concerned about any given one. Um, it's really that, that that layer that lets me integrate them together. That's my, um, my biggest push, right? So, um, years ago we, we used something called ops hub, but you, you had to script it out, and it was a lot of work to do that. So now, um, I, I use tasktop pretty, pretty much in this pipeline because it makes short work of being able to integrate these tool chains, right? In and out. But anything that would allow us to do this type of activity.

00:09:59

The other thing that we have to be very careful of, and I think you guys do too, is as we grow, right? We have to actually secure our factory, right? So one of the things that I spend a lot of time with my teams on is a how to build things like hacker stories. You know, before we used to think, ah, test driven development. Yes, perfect. Now I need to do security driven development as well, right? So it can't just be, oh, I compile, it's gotta be, oh, how would this be exploited? How can it be exploited? And in every step of that delivery pipeline. So integrating things like fuzz testing, right? Which, uh, injects kind of errors problems into your code security, vulnerabilities, penetration testing, right? These have to be a part of my delivery pipeline, otherwise, I can't actually deliver the right software factory, the government space, because most things that we build require a lot of security, continuous iterative development. I stole this picture lock, stock and barrel from the DOD Science board report. And they said, Hey, here's, here's waterfall up at the top. You know, then we got down to Agile, but somehow we just couldn't get it all the way out the door. And now this is how they refer to kind of agile and DevOps, right? Small single piece flow, bite size batches across the piece. So that's not how we've worked historically. This'll be a new thing for a lot of our, our, our government customers

00:11:36

Metrics. Again, I'm getting the T-shirt for Slack, even if they won't let me wear it to work. 'cause ugh, it has to go. You know, they're always like, oh, well, so and so had that much slack. I'm like, oh, that's brilliant. Were they writing in ada? Were they writing in.net? Did they have, you know, node js? What, what was this magical slack that you're telling me about? And I wouldn't even mind if you measured it, if you didn't call it a productivity metric, that would be okay. Um, so I saw a great presentation yesterday with the Dilbert cartoon, probably you guys have seen this cartoon before where, um, in Dilbert, he goes, I'm gonna go code me a minivan. Right? That's exactly what you're, you're, you're, you're incentivizing people to do. So what do we get looking at? Well, velocity over time, you know, velocity over time is, is goodness, because it allows me to look at some predictability data. Um, cumulative flow, right? Love, cumulative flow burn down. They called these specifically out in the, in the report. And then cycle time and lead time. So cycle time, lead time, meantime to repair. And then as I secure my factory meantime, to detect, mean time to attack, meantime to get back to a known state. All of these things are excellent metrics, all of them better than some of our historicals.

00:13:00

So we've had a variety of programs transition all at ding maturity levels, um, with a variety of lessons learned. But I thought I'd quickly just kinda walk you through a couple. All right. So F 30 five's making their big transition this year. They just have transitioned. They've never done agile. So this is new for them. They just started and I started training teams in the February timeframe. We got F 16. They've been doing, um, agile and DevOps pretty much since, uh, 2013. So they've got a lot of experience under their belt. And then F 22, they've been just the last few years. Aegis, Aegis is quite large. And I'll, I'll take you into each one of these. And then Orion is the next, uh, you know, space vehicle.

00:13:48

So I, I asked each of the programs to kind of give me some high level about things they were doing, and if they could tell me, you know, what were their, uh, key lessons learned. You're gonna see that throughout most of the presentation or most of the, the results I got back from teams, um, transparency and greater collaboration was our biggest return. That was the thing that they learned the best, right? So F 16, again, 15 to 18 teams. Now, I believe that this is just for one of our, our groups. So this might be the Taiwan retrofit. Um, we've got other ones as well. Um, what, what types of things they're doing, what types of metrics?

00:14:30

F 22, right? So they've increased, again, collaboration. They've really looked at moving to product versus project and using cross-functional teams, which is a, a different approach for us. Um, they've got a number of pair programming work cells. So, so great job for the F 22 guys, and they continue their journey F 35, right? So moving to cross-functional teams. Now I kid you not, this is a big thing for, for us to, to teach because I can go to any number of programs and they'll introduce me to their cross-functional software team and the cross-functional systems engineering team and their cross-functional test team and their cross-functional. I'm like, I feel like we're not quite on the same page with what cross-functional means. I'm just saying. So it's, it's something that we work on pretty regularly and prioritization, which is also something we're not incredibly good at. So they've, they've already started, they're pretty new.

00:15:32

And like, this is just a pilot start, meaning, you know, if once F 35 transitions, there'll be hundreds of teams. So it's, it's pretty large. Aegis Aegis has been doing this for multiple years. We have about 1300 people across eight release trains. There are about 157 different teams that, that deliver those, those capabilities to, to the war fighter, um, in all different types of technologies. Everything from ADA to sea, to, to Java and everything in between. Um, having a flow based approach was one of their key, um, benefits. And when I talked to, um, Christian Peterson, which is one of the folks that leads up the Agile teams in, in Moorestown, their New Jersey, he was saying his goal for this year is to, to be able to tell the teams even less, right? He wants to give them even less direction. Let them own the work even more.

00:16:32

Stop telling them exactly how to do things. So, um, I, I really see them getting better and better all the time. Orion, again, communication. So I remember NASA saying in one of our release planning events that they thought they had, had 12 weeks of, of conversation in, in that two day PI planning event, right? 'cause we operate, um, the scaled Agile framework with them. Now, I know that, you know, some people love safe. Others would say, Hey, it's too prescriptive. Um, but I, what I will tell you is my, my previous boss has a great statement about safe in the government space. He said that the scaled Agile framework is a gateway drug for the government to go agile. And I totally agree with him, right? You're not gonna go to a complete flow based approach without any structure from the amount of structure they have today. So for us, it's excellent. If you guys ever decide to look up the DOD chocolate chip cookie recipe, just so you know, it's about 42 pages. All right? So when I tell you that they, they may go above and beyond with the amount of precision they give you. I'm not kidding.

00:17:47

All right? Rethinking software sustainment. So everything is a continuous knowledge loop. Everything. So I call this our feedback highway, right? Not just for security, but for everything. But the big thing is really, you know, getting that intelligence back to the teams at each of those different areas. All right? So I, I focus specifically here on security, but the information, the data coming back to all of the different stakeholders, hugely beneficial for us

00:18:21

Investing in our workforce. So we have a number of communities of practice. We've had this for a while. Um, some are more or less active. Um, I have, so I, I have the, I'm the corporate community of practice lead for both the Agile corporate community of practice and the DevOps community of practice. We have two different ones, and I have about 800 members in the DevOps community of practice, but that's still coming up to speed. And I have about 1800 members in the Agile community practice, right? 'cause it's been out there a long time. Um, it's helpful. It's just not the end, right? It's, it's a good mechanism for communication. Uh, team safaris have been really helpful. Um, which is when a team is doing something really well, bringing another team to watch, as opposed to telling them, right? It, it seems to provide a lot more context and tacit knowledge.

00:19:13

Um, pairing right? This, this took a long time for us because, you know, I had a number of customers that said, I am not paying two people to do one job. Um, until they really started seeing the benefits of how much faster work could get done with this. Um, we have an improved tuition strategy. So before you could only go to school for, you know, your master's or your PhD and it had to be in these specific fields. Now they're, they're funding things like certifications, um, and they're even providing funding for things like meetups and stuff like that. Um, we have a variety of hackathons right now. Our current hackathon is a machine learning hackathon using the Spiro robots. Um, and so we're, we're using the robot framework and, and we have, um, different teams at eight different sites across Lockheed Martin and then Shark Tanks, right? So we're doing a lot to invest in the workforce, but it's, again, we've got a, we've got a lot more to go

00:20:13

Machine learning. Again, we have, um, a community of excellence for machine learning. We're doing a lot of work in this area for different things, autonomous vehicles, autonomous drones, all of that stuff. Um, this one where the DOD Science Board said for, you know, leveraging machine learning for IV and V, this is the only problem we've seen. And I don't know if exactly what the result or, or how we could maybe fix this is, um, is currently, it seems like it's not quite mature enough to do the job. And here's what I'm, I'm, uh, referring to, and maybe it's just because of how we're doing it. But what they found while they're doing this integration verification and validation and trying to do machine learning for it, that the, the, um, the software's learning something different in the development environment, then it learns in the test environment, then it learns in ops and they don't know what that difference is, which is making it difficult. Now, one could argue if we didn't have Snowflake environments, this would go away. Um, but that, that currently seems to be a barrier in this space for us. Alright. I got done early, which is awesome. 'cause usually I talk too much. But it's cool because you guys will ask me great questions. All right. So, so that's kind of my presentation next year. I can tell you what the impact of this, uh, DOD science board report is. Um, and also I can tell you that I, I support the NDIA working group to help build artifacts and, and things to enable our, our, our government counterparts to make this transition. Any questions for you guys? How about you?

00:21:59

I actually kind of have, so before I, uh, right, right now I've worked Silicon Valley. Before that I was an Army acquisition.

00:22:05

Okay?

00:22:06

So, um, the first question that I always kind of comes to my mind when I look at this is, um, I remember my Army program getting an award from the Army, not all that terribly long ago, five years ago. Mm-Hmm. <affirmative>, uh, for being the first Army software program to implement Agile, excellent, uh, with our, uh, two year software with East Side <inaudible>. Um,

00:22:29

I, yeah.

00:22:30

Joining Silicon Valley and I, I I, I have to laugh at it. Mm-Hmm. <affirmative>. Um, but you know, really what we got the award for, when I look back on it, you know, we weren't really getting the award. Well, yes, we did get the award for reducing the software release type out in two years, but really we got the award for changing our contracting strategy Mm-Hmm. <affirmative> to support it. And I guess when you look at trying to go as far agile as, as this field been signed for, is trying to say, like, my first instinct just says, do you actually get the sense that the, you know, as a contractor, right? I often felt that the, uh, way my contract was written did not actually reflect what the government seemed to expect me to do when it came to, uh, implementing Agile. Agile,

00:23:16

Yes. There's still some communications things that we're gonna have to work on. I mean, one of the things that I would recommend, and, and I've even recommended it to, to people that were here is the DAU or the Defense Acquisition University is a great program. However, even though they're making this change, they've only got one class in DevOps one, right? There's a lot of work. And, and they could really leverage a lot of that experience from the commercial space to bring that into defense acquisition university or even maybe take advantage of things like MOOCs or, or other online training. But there's, there's still a lot of, um, knowledge that has to be given on the customer side too. Otherwise we're speaking two different languages.

00:24:01

Other question I had sorry to ask too. Yep. The other question I had was kind of on the sustainment side.

00:24:05

Okay.

00:24:06

Um, are we actually seeing, do you think we're actually seeing a shift in the defense culture? So certainly I know that every program in the Army Mm-Hmm. <affirmative> was structured and I believe that the Air Force and the Navy were doing it basically the same way, was structured specifically there was a difference between the acquisition branch and the sustainment branch Yes. Of the same program.

00:24:28

So I volunteer a lot for what's called the government acquisition organization or the GAO. And the big conversation that we've been having is the color of money, which is exactly what, what you're talking about. This problem has to be resolved. It hasn't, right? Because once I start doing things like DevOps and I'm getting into continuous delivery, when does sustainment really start? What does that look like? And if software really is immortal, you know, it's Yeah.

00:24:53

Mm-Hmm, <affirmative>, yeah. No, we have to do some very interesting gymnastics around the color of money to do anything even vaguely. I,

00:24:58

Yes. That's definitely a barrier. Yep.

00:25:01

If I may, I mean, mm-Hmm. <affirmative> you, um, you ought to look up the DSB report because there were I think, seven areas that they were asked to look at in those words. The two questions you had were two of them. Perfect.

00:25:13

Yep. Absolutely. Anybody else? Yep.

00:25:18

So I lead the center of excellence with my, within my organization, and we've done things like use success stories and gotta great ourselves like video crews and, you know, really selling and evangelizing. But you mentioned teams to bars as well. Can you expand a bit more about like what's the structure and how do you

00:25:34

So what I found is it's really hard to tell people what they're doing right, or what they're doing wrong, right? Because everything they look at is with your lens. And, and actually I got this from my, my friend Suzette Johnson. I didn't invent this, so I was like, oh, this sounds like a great idea. 'cause I was explaining to her how difficult it was to kind of demonstrate. So, um, what I did was take her idea and, um, we've got teams like I, I have lots of teams, let's say in Morristown, New Jersey. I have lots of teams in, um, uh, Denver. If there's a team, especially in the same facility, doing something exceptionally well. And, and you know, their coaches are kind of saying, you know, or the, the other coaches are kind of saying, this is a problem area. We're bringing the other team to visit a just to say, you know, what to, to, to look at how this team runs a daily standup or maybe sit in on an inspect and adapt workshop.

00:26:27

Um, the one most interesting thing I found is we're really good at picking out what somebody else is doing wrong. So it usually turns into a quite a good two way feedback. 'cause they're like, well, I don't know why they're doing that. You guys are too. I'm just saying. But yeah, so, so basically we've just go walk over and, and, and watch 'em. Um, and then it also does something interesting too, which is, uh, they start talking to each other. Um, you would think that would be a normal thing if we're all in one facility. But it's, it's not, if you're on a different team and a different program, we almost think that you're, you know, in Jupiter. So, so it's, it's provided a lot of benefit from there too. Anybody else? Yep.

00:27:10

You showed different teams doing slightly different things. Can you talk about why that is? So why you gave the freedom for those teams choose as such?

00:27:19

So I think that it's gonna be really important, even though we want alignment, right? To ensure that teams can have freedom to implement what works for them. Um, I come from a culture where we try to make one size fit all, all the time, and it's just not true. Um, you know, it's a difference between working with my multimodal biometrics team that's got maybe 12 to 14 teams in three different locations versus working, let's say with Aegis where they're all, you know, either in Morristown or Syracuse. It's, it's different. They have different problems. So the things that I wanna standardize on are things like cadence and synchronization, maybe, uh, focusing on business outcomes or objectives and key results. Um, and, and maybe some of the best practices like that. But other than that, I truly believe for each of your programs, you kind of have to build your own lightsaber. And so I think it's more important to let the teams own that work and, and do it the way they think would be best. You know, it would be like me telling another coach to operate this way. Well, you can't operate exactly how I operate. I'm different, right? So what works for me isn't gonna work for everybody. Yep. Let's

00:28:32

A follow on from that. Um, what would you, how would you say you're dealing with some teams, and especially in government organization, we've had people who've been there for years, and you come up against the people who say, look, we've done it this way, we don't want to change. And they're like the people who want to hold everyone else back. Mm-Hmm. <affirmative>, how do you manage them and get them on board with your dream so that everyone's pulling in the same direction?

00:28:56

So I don't always make progress. Sometimes I, I totally have run into that. Um, but on other cases, and this is, maybe this is maybe manipulation, I don't know. But anyway, I usually approach every team as I know this won't work for you. I know that you're special and you're unique and your problem isn't like anybody else. 'cause that's how I know I'm at Lockheed, if that's what people tell me, right? So, so I start by that and I'm like, so if we really wanna get people off your back, right? 'cause they think there's something to this, how about we just pilot it and you can show 'em how none of this works. I go to grins and giggles, just follow a verbatim, let's see what we get, right? Which overcomes the barrier to, no, it won't work, because they can prove that it doesn't.

00:29:35

Um, once that happens, once, once that initial barrier goes, you'd be amazed at how actually amazingly smart the folks we have on our teams are and how they can make it work. So once you take away, you know what, I know this can't work in this environment 'cause you're special, um, all of that kind of argument seems to go away. That being said, I've still had teams that were just like, no. Um, and that's fine, right? It's, it's not about being DevOps or, or agile or, or anything. It's about delivering better business outcomes. So however you get there is totally fine with me. All right. So my time is up. Um, hope that was helpful and I really do appreciate you guys coming. 'cause like I said, there's some great speakers in this block. All right, thanks.