Architecture Driven AI Engineering

Host Intro (Gene Kim)

The next speaker is JD Black, Chief Engineer at Northrop Grumman. They were founded in 1939. They have over 43,000 engineers building some of the most advanced aerospace defense systems in the world, including the James Webb Telescope, the $10 billion satellite that has dazzled scientists for almost a decade. Which, by the way, thanks to JD, I was able to tour the facility that manufactured it along with Jason Cox from Disney and Kent Beck. And I guess this underscores the value of having the right friends — so all of you in the room included.

I first met JD in 2017 because of his heroic efforts to try to bring modern software engineering practices into his industry — not just his company, but across the entire defense ecosystem. In 2023, he gave a presentation here at this conference called "Architecture as Code," where he merged not just DevOps, but also a very arcane field called systems engineering, kind of famously known for staying in their ivory tower, but very much needed when you work in the most demanding engineering environments where the stakes are so high.

One of the things that JD told me in the last year was how much fun he was having on weekends working with AI on his hobby projects. And he said his mission is to make sure that when people in his organization go to work on Monday, they can use some of those same technologies for the most important work in the most consequential environment.

So here is JD to talk about his mission. Thank you, JD.

---

Jondavid "JD" Black

Thanks, Gene. All right. Good morning, everybody. I'm thrilled to be here. It's been a fantastic couple of days so far. We're so excited to learn from all of you, and hopefully we can share a little bit of what we're doing.

The last couple of days, we've heard the word "rigor" a couple of times. I'm going to take that and dial it up to 11, and I think it's something that matters to everybody in this space.

I'm JD Black, as Gene said — about 26, 27 years in the industry at this point. I've done government contracting sort of from day one. About 20 of those years I spent doing missile defense, and for our CFO friends, that career has been about just over $12 billion of growth that I've had the opportunity to contribute to in our business. Lots of fun stuff. A couple of years ago, I did kind of join the dark side and went to corporate, really looking to bring some innovation and speed to the game in that space.

Some people don't know who Northrop Grumman is, so let me give you the quick overview. We build really cool things. If you need to fly around the world and not be seen by anyone, we know how to do that. If you need to shoot a bullet with a bullet from the other side of the planet, we know how to do that. If you need to look to the beginning of time, we know how to do that. We do a lot of fun things. We helped put the astronauts from the Artemis mission around the Moon. Lots and lots of cool stuff. We're about just shy of 100,000 people, so pretty large. Scale is a real challenge and something that we're excited about working through.

It's a little bit intimidating with Mick in the room, but I tell my teams that we really only have two fundamental constraints: physics and economics. Now, in my previous life, physics was a very real problem. Things like speed of light, things like curvature of spacetime, inverse R-squared — these are all things that have had a meaningful impact on the work that we do. In corporate world, it's all economics, right? It's all about driving effectiveness and efficiency through what we do. And Mick and the research that he's done has really helped us codify what that looks like.

We understand the construct of these revolutions. We've lived through them. We understand the shape of them. And right now we're really in that creative destruction phase with AI. I like to look at the last cycle that we all went through as DevOps came into the space — there was a ton of hype, we all saw it happen, and we've also seen it kind of settle into that wealth generation phase. There are new leaders in the market that are really driving things forward.

When we look at where we were focused in our fundamental lifecycle, we were focused on product delivery. How do we streamline and optimize product delivery? How do we drive predictability in that? How do we reduce variability in that? And it transformed how we do work. We can't imagine ever going back.

When I look at what's going on now in the age of data and AI, we're really addressing that point prior to commit. Now we can go and look at optimizing product design and development. Comprehending that — and recognizing that what we've learned in the last cycle still applies to what we do today — is so important. So much of what we saw yesterday, I really appreciated the patterns that Randy showed. It really reinforced the things that we know work well. Turns out they still work well, and we need to think about how we adapt and tailor those things to the world we live in with the new technology that's coming to bear. How do we optimize the fundamentals of what we do on a daily basis to take advantage of this without compromising on things like quality and capability?

---

I kind of think of what we've seen over the last 18 months. Who remembers about 18 months ago when all of us didn't have a job anymore and we were going to become prompt engineers? That was real. I remember I was driving around DC with Gene and Kent Beck, and we were geeking out on how cool Copilot in development was, and it felt like the thing that we had been missing.

And then it just continues. We keep seeing — what is that next thing that comes? What is the next thing? And we're continuing to see this happen. Wasn't that a couple of months ago we saw agentic engineering? A couple of weeks ago, we saw a guy say AI is going to destroy human civilization. I tend to be a little more optimistic than that.

But every step along the way, we've been thrilled and excited. We've seen a step forward, but we've also seen that there's still something missing. And that's really what I'm trying to focus on — to understand what is that missing thing that we can bring in.

Hopefully I can at least get us on a path that I think is probably where we're headed.

The challenge I see, I call it the Anchorman problem, and hopefully you guys see it too. "60% of the time, works every time." The challenge I've got is for the kind of stuff we do, 60% is not good enough. This is real mission-critical stuff. People's lives are at stake. I can't live in a world of hope. When you work for colonels and generals for a number of years, you can very rapidly take on the mantra that hope is not a plan. And too much of what this technology is doing is just — we're all learning to trust the machine and live on hope. I think there's danger in that, and I think we can do things just a little bit differently to help overcome that problem.

---

So again, I go back to what we've learned over the last 20 years. We learned that optimizing people, process, and tools drives value in what we do. It drives efficiency in what we do. And you can't skimp on either one of those things.

The fundamentals — when I think about the DevOps movement and everything, I go all the way back to the work that Jez Humble and Dave Farley did. Continuous delivery to me is the center of it, and having good CI practices and just making that normal table stakes of doing business. It was a big shift for people. It was threatening to a lot of people, but it drove so much value. It streamlined what we could do. It made us more effective in the space. We could attack things and do missions we'd never thought possible before, at pace we'd never thought possible before.

Understanding good process — we all have our own processes, we optimize them for the work that we do. That doesn't change. We need to know what that is. I'm a huge fan of flow engineering and the work that Steve and Andrew did. Understanding lean still matters, and we need to get to that. And then of course, finding the right ways to bring your people along. We saw a lot of discussion yesterday — what is the human role in all of this? We're all learning that together now. It's going to be a little bit different, but we're not taking humans out of the equation. We've got to figure out what that looks like.

But we also have to translate what we know into how these machines work. The analogies that I see really center us around the idea of agents, workflows, and skills. It parallels people, process, and tools almost beautifully. But we find that we build them incorrectly too often.

What I'm learning — and what our teams are doing — is learning that agents are your subject matter experts. You need to keep them really good at one thing. We all learned: don't be an I-shaped contributor, be a T-shaped contributor. I really like my agents to be I-shaped contributors. Their job is to know what good looks like so that they can move forward with that. But I don't need them to be so deep in the "I" that they can't generalize a little bit. I use skills to do that. I don't put anything tech-stack specific in my agents if I can avoid it. The skills are the things — the tasks. How do you do a given task? Let the agent know what good looks like, and then let the skill tell it how to do a given task in your context, in your space.

And then one of the big missing pieces on my mind is workflows. I see it a lot. We see skills that are bulleted lists of things to do. I see agents that are a bulleted list of things to do, and I'm right back to the Anchorman problem. I cannot get my agent to follow my bulleted list consistently, and I certainly can't demonstrate that my agent did what I asked it to do consistently. And so what we're doing is pulling workflow out as an independent construct and establishing clear, predictable, atomic behavior — with determinism and exit criteria validation in there as we go.

The other piece of the workflow that's so important is understanding that sometimes it's human in control — well, it's always human in control. Sometimes that's an in-the-loop thing, sometimes that's an on-the-loop thing. It really depends on your context, the criticality of the risk that you're dealing with in your space. But we go back to the principles that lead us to how to optimize this stuff. It's what we already know: value stream management, good configuration management — so important. Good ways to validate behavior and resiliency and repeatability in that.

---

When we look at what we're doing, the whole construct of outer loop and inner loop really resonates with me. I like to think of the outer loop as understanding and delivering an executable strategy. How do I get from point A to point B? And that is so much of what we have to deal with in our business. When you're asked to do something that the world has never done before and we're not sure is actually possible, you have to have a really good strategy on how you're going to approach that. You need to understand not only the features that you have to work through, but the risks that are facing you on a daily basis. You're going to make decisions and build tech — and be conscious about that, have it part-visible in your plan. And then be able to translate that into something that teams can go and execute, that they can get their hands on and take advantage of, and feed that into your iterative execution loop.

We already know that small batches are optimal. This comes straight out of the work that we've done in lean. So much of DevOps is built around that, and it continues to be true.

We've also lived through the agile revolution. I'm a huge fan of agile and most everything associated with it. I tend to frame agile as 95% communication and 5% process to force you to communicate better. If you're doing that, you're doing agile pretty well in my book. But it's really about breaking that work down, understanding the dependencies. What has to come before what? What do you need to learn along the way? How can you pivot and work through things effectively? And it still applies here. Build that structure in your outer loop, and then your inner loop should be small and targeted.

I get accused sometimes of saying, "Well, you're just implementing waterfall in your agent cycle." And to some extent that's true, but if my waterfall lasts a couple of hours or a few hours, is it really waterfall? It's kind of what we're doing. But I can lean on the agents so heavily now.

---

We've adapted our normal process just a little bit. We start with systems engineering. Gene talked about 2017 when we really first got together — we were doing a workshop, and I set up one of the workshops to say, "How can DevOps optimize our systems engineering process?" I'm fascinated by this. And of course we started the workshop and Gene's first question was, "What do you mean, systems engineering?" I'm like, "Okay, here we go."

Systems engineering is a critical thing. We don't get to the Moon without systems engineering. It's just not possible. You can't get there. When you're doing really complex stuff that really matters, you need to understand the problem. And systems engineering lets you frame that out. When you know the problem, when you have that depth of knowledge, then go look at what the solution space might be. But all too often we dive straight into, "I can implement that with my agent in ten minutes," without truly comprehending the problem and understanding what we're doing. Systems engineering is a great way to do that.

We are seeing that open standards are incredibly powerful in the space. We saw a brief yesterday — Cal talked about what they're doing with CALM in his space. I think it's phenomenal.

But there are true open standards that exist today. OMG, last September, published something called the Systems Modeling Language version 2 — SysML v2. It's all code-based. And guess what? Your agent can work in that space. It can read it, it can write it. It's also verifiable and validatable. I can lint it and effectively compile it. I can capture full traceability from my desired visions and objectives and goals down through the processes that I have to operate through, step by step, with clear understanding of my requirements.

I can structure that in a way that lets me get to architecture review, and it doesn't have to take weeks or months like we have historically. It takes minutes. And I get objective evidence — I can say, "Yea, verily, this is what my system is and what it needs to be and where we're going." Then we go into design. What you're all used to, right? The user experience still matters. By automating this process, I can inject things that we often have a habit of overlooking. I want to understand what does my reliability, availability, maintainability look like if I do what I just said. What's the impact of that? Does it affect me positively or negatively? You can incorporate that stuff right up front in your flows, and I would encourage you to do it. It's the principles of shift left — the things that I need to do all the time. I shift them as left in the process as far as I can, and then I automate it, and I review it.

One interesting thing we've done — normally plan is the first thing you do. We've shifted plan out. Why would we do that? When it only takes minutes to get through your systems engineering or design changes for your given feature, and you can do it in a way that is version-controlled and text-based — it's part of my Git repo — I can diff my main with what I just did, and my agent can build a path to get from point A to point B incredibly efficiently. It's amazing how well that works, and I would encourage you to try it.

Then we go into normal things. We go to implementation. Never start with implementation — understand the problem first. But once you've gotten through that, get to your implementation and put the structure and controls in place that you know you need, that we all know is necessary to get the job done. And then finally, we can go into integration and release. Following a simple cycle like that — it's stuff we all know — but how do we shift left on the pieces that we know we need to, or often skip in the flow? And then how do we automate it and put just enough human in place so that we have the accountability and the rigor that we need in that flow?

---

I have two things that I think — and I've kind of hit on them a little bit — two things that to me are the missing pieces so far in our agentic journey.

Thing number one is architecture. Architecture is not PowerPoint. Architecture is not a sequence of meetings. Architecture is a concrete thing that defines your problem, defines your approach, and defines your solution. It's fully traceable. Know what your requirements are. I was so thrilled to see EARS requirements mentioned earlier. EARS is a phenomenal way to capture requirements because it gives you a structure and a mechanism that you can directly trace into automated acceptance tests — and that's exactly what we do. We know what our requirements are. We evaluate them against INCOSE quality standards. We structure them using EARS, and I can translate that directly into a Gherkin file with a tag that traces it back. Now I can have a script that says, "Do I have coverage of every requirement I know I'm accountable for in the system?" It's a yes or no question, and it's incredibly powerful.

I can look at all the different elements and other concerns. Do I know what my interfaces are? Do I understand the flow of data across my system? Do I understand my performance requirements, and can I measure them? Safety and security matter. Environmental concerns matter. But you can capture all of this. And I mentioned earlier that SysML v2 standard is a really good way to do that. It is an architecture-as-code structure. I talked a few years ago at the DevOps Enterprise Summit about architecture as code and some of the work that we were doing, and I'm thrilled to have abandoned that project now because it is an industry standard and it's phenomenal.

What we're seeing now is we used to spend a lot of time — boxes and lines, boxes and lines, dragging boxes and lines. Now our agents can do that work for us. Our people are not supposed to be writing lines of code or dragging boxes and lines. I want our people making engineering decisions to make sure we're achieving the outcomes and objectives that our organization and our customers need.

So first and foremost, architecture matters. We've known this all along.

Secondly, process matters. Workflow matters, and it needs to be explicit. We've built out a structure within our harnesses that allows us to define two fundamental types of steps: agentic steps and human gates in the process.

Agentic steps are interesting. I can break everything down in ways that I can ask my agents to be incredibly precise and specific at very focused tasks, and their performance goes up. You guys have probably seen it — I can ask my agent to do a broad thing, and you're right back to the Anchorman problem. Sometimes it works and sometimes it doesn't, and it's your job to know if it did it right or not. What we find is breaking that down into small steps really does matter.

A great example: if you look at our actual workflow that we've implemented, look at phase four — phase four is just implementation. Step one of this was "implement the defined architecture and use test-driven development practices." Well, it turns out that just didn't work well. More often than not, it would skip testing. Like, how do you skip testing when your agent is directed to do test-driven development? So we break it into three to six steps. First, define your tests — and then we have automated exit criteria, deterministic scripts that I can run to know that I accomplished what I intended. Do I have tests that fail? It's a yes or no question.

The other thing that's really interesting here is it's auditable. Our harness and the structure that we put in for this will tell you exactly how long each step took, what agent did it, what human was involved in what gate and did they sign off, how many tokens were used, how much did it cost, how many tools were called. So now you get back to value stream mapping, and the hardest part of value stream mapping is pulling the measurements in. We've automated the measurement collection, and now I can understand where my bottlenecks are quantitatively and start to optimize that. I think it's incredibly powerful what's going on there.

---

What we've learned kind of amplifies what we've known already. It's just tailoring and applying it to the domains that we all operate in.

First and foremost, we all feel it — everything's about speed right now. That's what the market demands. Speed, speed, speed. For us, we achieve speed through rigor. That's what we're going with, because we cannot compromise on quality. And understanding how to build that rigor into your process where you embrace speed is so important.

Small batch execution — absolutely critical today. Refine and focus the technology to do the job you need, in the sequence that needs to be done, in a verifiable way. Maintain human accountability in that process. Make sure you are building people who understand how to make engineering decisions rather than click buttons and drag boxes and lines. Engineering decision-making matters even today, and it will not go away. Nobody's losing their job if they're credibly making quality engineering decisions. That's what we need in the market.

And then architecture matters so much. Understanding your architecture, putting those pieces together, connecting those dots, showing your flows, measuring it — it is incredibly powerful.

We've been able to take this and apply it to the work that we do. One thing that I'm incredibly proud of — Jamie is in the room today. We've gone and used these approaches to build out replacements for COTS tools. We spend almost $30 million a year on a given COTS tool, and he's been able to go and build a credible replacement that integrates directly into this process in a matter of weeks. It's incredible. We're starting to see full-stack infrastructure — we can build it all out in a matter of days. We can deploy full integrated ecosystems in minutes, and we're doing it with a team of seven people. It's unbelievable what you can do.

The challenge is not skipping something. Don't skip the stuff that's important just because the technology tempts you to do it.

My ask for you: help me embrace architecture to drive these agents forward. Build stuff up so that there's more data for the agents to train on in the next cycles. It works, it's good, but it takes a lot of guidance right now.

These standards are really simple. Version control — oh my God. The next time I open my harness and magically get updated skills and agents, I'm going to lose my mind. But these things have a real impact on how we work, and they should be semantically versioned and controlled as dependencies. If you work in the domain of building repositories, please, please stand up and build that for me.

Standardize agent workflows. I don't know exactly what this is going to look like, but steal all my ideas and build something that we can all use together.

And then finally — come up to me afterwards. We've been talking so much for the last 18 months about what do you do when code is free. I want us to think about it a little bit differently: what do you do when rigor is free?

Thank you.