Minutes Matter: Fielding Edge AI for Personnel Recovery in 6 Weeks

Host Intro (Gene Kim)

All right. Early this year, I got to hang out with a bunch of switched-on engineers and technology leaders at Rise8's Ship Summit, and that's where I met Kevan Mordan, one of their delivery leads. We were talking and marveling about our lives, how it's changed because of agentic coding, and we were joking about what was going to happen when we can shave off two orders of magnitude from so many software tasks. And then he said something that actually stunned me.

He said, "Yeah, it saves time, but we've never worked harder." He then described to me an amazing project they did for US Special Operations Command in six very intense weeks. During this project, they worked on a super ambitious project that they could have never undertaken without AI. He slept under his desk on many nights, including on his birthday, and he still marvels to this day at what they built.

And so if you've read the book Team of Teams by General Stanley McChrystal and Dave Silverman, who actually spoke at our conference in 2020, you already know of SOCOM. But for those of you who haven't, they are the force in the US military who they call when missions cannot fail. They're the most selective, the most trained operators in the world, doing missions like rescuing downed airmen or executing hostage rescue missions when nothing can go wrong.

So to tell the story, Kevan is co-presenting with Kenny Slater, a retired Army Master Sergeant who served in Special Operations communications, including supporting the Commanding General of US Army Special Operations Command.

So here is Kevan and Kenny.

---

Kenny Slater

Thank you, thank you.

My name is Kenny Slater. As Gene Kim said, this is Kevan Mordan. Also, as Gene Kim said, before I start, I want to say thank you so much, Gene Kim. This is my first Gene Kim summit. It has been nothing short of amazing. All the presenters so far have been extremely informative, and it's been awesome to watch so many brilliant minds come in one room and talk about a subject that not only affects a lot of our careers but affects the world as we know it. So thank you. This has been awesome so far.

Today, we're here to talk about how Kevan and his team were able to integrate AI into some mission-critical military tooling. Not only did they use AI to update this mission-critical military tooling, but they used AI to integrate AI into this mission-critical military tooling. It is some of the most important mission-critical tooling in my opinion because it helps with certain missions like personnel rescue.

My background, as Gene Kim said, I am a retired US Army Master Sergeant. I know that I look like I'm 12 years old — you're going to have to trust me on that one. I have five combat deployments, four of those in Afghanistan. I have about a little less than four years total deployed in active combat zones. Three of those working directly for US Army Special Operations, and two of those directly attached to US Army Special Forces Green Beret teams. So the majority of my career I spent supporting Special Operations.

The hypothetical, but too common, scenario we're here to talk about today — it's a time-critical catastrophe in which not only every single minute matters, but every single second could be life or death for the US military's most valuable asset: the US service member.

A pilot goes down in enemy territory. It's a very unfortunate situation, and it's an unfortunate situation that's happened a few times recently. The pilot ejects, lands somewhere below, potentially injured, alone. I don't want to put words in their mouth and say they're scared. I know a 12-year-old baby-faced Master Sergeant Slater would be scared. From that moment, there's a clock running.

We have an estimated location of that downed pilot — if we're lucky, usually never an exact one. From that moment, every minute that passes, that area widens and makes it more and more dangerous to not only that pilot but the team that comes in after them.

This is personnel recovery. It is one of the most time-critical missions that we run. In the Army, they have this thing called the Soldier's Creed. If anybody here was Army, they know exactly what I'm talking about. It's basically a creed you have to memorize, say over and over and over again for basically your first year of military career. And in that Soldier's Creed, they have what they call the Army values. It's a set of core principles that they believe every soldier should live by. And in those core values — I'll give you some of them — a few of them are: "I will never accept defeat," "I will never quit," and "I will never leave a fallen comrade." And I promise you, that last one, a lot is at stake personally. So personnel recovery is near and dear to my heart.

Everything you'll hear over the next 20 minutes comes back to one question: how fast can we get the right people in the right place with the right picture?

A recovery isn't just one aircraft. It's a complex environment. There are multiple helicopters, there are other aerial assets overhead, and there's a ground team below, and they have to work as one.

The ground team, they come in, they secure the area, they treat any injuries, they get the pilot out. The helicopter — they see all this from above. They see rooftops. They see open areas. They see roadways. The ground team, they see something different. They see walls. They see doors. They see alleys. These are two very different pictures of the exact same place.

To coordinate at speed, both have to point at the same building and call it by the same name, with no room for error.

That shared picture is what we call the GRG. The GRG stands for Gridded Reference Graphic. It's what Kevan and his team took from a very slow manual process and turned into magic, as far as I'm concerned.

So yeah, GRG — it's overhead imagery with a grid laid over it, as you can see in the middle picture. It's a gridded overlay on a graphic, and in that gridded overlay, there are buildings that get numbers. So everybody can point to the right building and say, "Hey, our guy is in building 205." So everyone knows exactly where that is. Quality and detail matter because it's what people see nearby.

So this is how building a GRG used to look. We used to do it by hand. You would have to label building A1, building A2, building A3, A4, 5, 6, 7, 8, 9, 10, 11, 12, 13 — you get the point. It's a lot of clicks. Honestly, this is the easy version. I'm pressing a button.

In the field, the operators — they have a handheld device. They have to zoom, they have to tap, they have to click, they have to label. Then they have to zoom, they have to tap, they have to click, they have to label — zoom, tap — over and over and over and over again. And a lot of times, there isn't enough time to finish it. They have to get out when the mission requires it. So if they have to go before the GRG is ready, crucial detail could and probably will get missed.

Every minute building a GRG is a minute not preparing for the rescue.

Our team builds GRGs in a tool called TAK. TAK stands for the Team Awareness Kit. There are a few different versions. The one they work on, it's a handheld device, and it's what the ground crew already lives in. But look what it runs on. Android phones that are often a few generations behind, like the S20, S23 — I think the S20 came out in 2020, so you can imagine how old that is. And Windows laptops that are not getting any younger. And additionally, they have a lot of security and compliance software on them, which makes them slower and slower and slower. Small screens, old hardware — it all has to work completely offline. I think it's fair to say, where a lot of these teams go, there's not going to be new deals on AT&T Fiber.

So we had a slow manual process on small, aging devices with no network, and a team doing labeling when they should be preparing for a rescue.

So here's what we asked for. I need somebody to build a TAK plugin that makes GRGs faster. That's reasonable. And use AI to detect and label the buildings. Fully offline, no cloud, on the older Android and Windows laptops that you just saw, with full interoperability to TAK and a bunch of other acronyms. And if you don't know what any of these acronyms mean, that's perfectly fine — neither did the developers that we handed this project to. Meeting government cyber compliance, which, if any of you have ever worked in the government, you know that government cyber compliance is harder than a personnel rescue mission. But I digress. In eight weeks. Actually, seven. Actually six, and our budget can only afford six people.

Let's strip all that away, and the requirement is simple: give the operators back the time they need to prepare for a rescue.

Kevan, over to you.

---

Kevan Mordan

All right. Well, thanks for that, Kenny. Wow, it's quite the ask.

So quick show of hands — who'd be willing to say, "Hell yeah, sign me up to do this project"?

That's great. I really applaud anyone that's willing to dive headfirst into this.

As Kenny mentioned, I'm Kevan Mordan, a delivery lead at Rise8, where my whole job is to deliver rapid and repeatable outcomes in production. But before this, I actually started my career as a cyber command and control officer in the Air Force, supporting air operation centers. And one of the many duties for that was actually ensuring that the JPRC — the Joint Personnel Recovery Cell — had consistent, uninterrupted comms for personnel rescue. So this is also very near and dear to my heart given my background as well.

But as we were trying to wrap our heads around the problem, this was a rough day-one checklist of what was going through my head.

The first one — as Gene Kim likes to say — assemble a team of switched-on people: people that love the challenge, they learn quickly, and they want to own the outcomes. Check. Easy enough. Buy S20, S23 phones. Well, we found some certified refurbished ones because they're that old, but check. Buy Windows laptops — no Alienware gaming-spec ones, unfortunately, but ones that match the specs in the field. Check. And that was really important to us because we could have probably developed a better, flashier capability on those, but that's not what was being used in the field. So if we did that, we would've shipped a demo or a proof of concept, not something actually usable.

Oh, wait, there's one last thing. Copilot. Check.

And the reason why that last one's important is because, to be honest, our team had some gaps to be able to do this in six weeks.

Native Android/Kotlin — hadn't touched in forever, like ten-plus years. WPF on .NET, .NET 4.0 — never done it before. I don't even know what WPF stands for. It's like Windows Presentation Framework or Forms, something like that. Also, .NET 4.0 didn't support ARM. That's okay. TAK plugin internals — we had to learn as we went. A computer vision training pipeline — never set one up before. YOLO, YOLT, Mask R-CNN, UNet — you only live once, so carpe diem. Edge runtimes, PyTorch, ONNX on Android 13, S20s, S23s, Windows — never had a care before. We did a custom .NET to Java JNI bridge because the .NET 4.0 framework was that old. Didn't know how to write one. Probably shouldn't have. We did it anyways. Complex path traversal math — not our day job. But at the end of the day, it still needed to be solved, and we had six people willing to figure it out.

And the cool thing is, with AI, sometimes that's all you need — because you can take a team of strong engineers and, with AI, they can get productive fast in stacks and frameworks they haven't touched in years, or never at all.

AI took all that nitty-gritty implementation detail and abstracted it away so our engineers could focus on how to solve the mission problem at hand.

And so now I'm going to cover what we spent our time on as humans, and the first engineering problem might not be what you expect. It wasn't code — it was a Slack thread. What is a building, anyways? I don't know. Twenty-four replies, it's still open.

And this is actually a problem because — is a garage a building? Some people say yes. Is a gazebo a building? Some people say yes. Is a tent a building? For some reason, the Army says yes. Kenny, please explain that to me. But the point is, if you ask 24 people, you'll get 24 different definitions that change day by day, whether there's a full moon out or Mercury's in retrograde. And when you train an AI model, you don't need a mood — you need a rule.

And what makes this harder is, from overhead, when you look at imagery, we had a thing out in the wild called elusive building mimics. If we take a look on the left-hand side with the low-res imagery, I don't know what's a building or not. There are concrete pads. Some are trailers. Are they semi-trucks? Is there a tower? Shadows? I have no idea. On the right-hand side, this one's nicely labeled, but there are concrete pads with very defined shadows. There are buildings under construction. How built does a building need to be before it's considered a building? These are existential building questions we've all asked.

And so what we ended up doing — we actually went down a whole rabbit hole where we did very high-resolution site imagery compared to low-resolution, to see if we could find any sort of artifact tracing back and forth. We scrapped it all. It didn't make sense because at the end of the day, what matters is the imagery on-hand in the operator's phone. So that's what we focused on. And we sat down and said, "Hey, what is a building to you?" And the simple definition is: if it walks like a building and talks like a building, it's a building. So we don't care in the real world if it's a building — if on the imagery it looks like one, because more often than not, for the pilot overhead, that's what matters. And it's easier to delete or have extra labels than to miss a label on the building that mattered at the time of the mission.

But now there's another problem. Any imagery on a phone offline — there's location, resolution, color, angle — all of these things directly impact training and performance of the AI model, and there are also ways of pre-processing this data. So what does everyone do when presented with a challenge? Well, we ran an AI bake-off. We trained four different AI models with various ways of pre-processing to see if we could actually implement this rule of: if it walks and talks like a building, it's a building.

And so here are the results.

YOLO — you can see it did all right. It missed some buildings. YOLT did a lot better, but guess what? YOLT takes three to four minutes to run on an S20, and it's not even that big of a model. Mask R-CNN didn't get much better. UNet gives us a good pixel mask, but it doesn't give us the actual building bounding boxes that YOLO does. And so what we focused our engineering time on here is: what is the right model, and the right way to apply the model, to get the right amount of building detection so we can label them? But there's also something else missing — these are just boxes and masks. Where are my pretty labels of A1, A2? The model doesn't do that. Okay. Next problem to figure out.

How should labels read so we could write an algorithm to write them that way so that people read them as expected? So if you take a look at this — how would you expect labels to flow? I don't know. So again, we sat down with the operator and asked, "Hey, what is your process here?"

They roughly went in and grouped buildings out — their first way of kind of setting the stage. And then afterwards they went left to right, as most English speakers tend to do. They tried to follow roads and not jump around, to have some sort of pattern of expected flow, generally in this direction. And then we took that and tried to encode a specification — all those patterns, all those clues — and we wrote an algorithm and tried to do our best. Again, it was hard because the operator was looking at roads, terrain, ditches, trees — everything that influenced it — where we just had a bunch of centroids. And so we tried to encode all that in math: the average distribution of points, clusters of buildings, line of best fit for the three previous points we were at, and how much angular momentum we were departing from on the next joint. But the cool part here is I didn't have to figure out any of that math. I had to specify it. All those AI training models, all the pre-processing algorithms — I didn't have to do the actual implementation myself. I had to focus on what mattered, and that was what mattered to the operator and to the mission.

And so now that we can get a full GRG, it comes back to our North Star. My whole purpose for six weeks of life was: how quickly can we get an operator a GRG out? Select an area of interest, get one auto-detected, auto-labeled, but then also have the ability for them to go through and edit, refine, change it — because we can only do the best we can with a draft. They have to own it.

And as Kenny showed on the display, these are not the most user-friendly interfaces. For any designers in here — preparing for a personnel rescue mission is one of the highest cognitive load, highest stress environments you can imagine, so the interface has to be simple and it has to work.

But beyond that, it doesn't just have to work — it has to work well.

And so this is what we chose to own. These are all the engineering problems we focused on. And because of AI, we had less legwork to actually do ourselves, and we could focus on the mission. So all of those mechanics, all these really complex things that would probably take me a PhD to get up to speed on, AI was able to handle all of the nitty-gritty implementation for.

And then what's cool about that is, because none of this is actually the mission — none of this is the outcome, these are outputs — they're very important outputs, because together they combine to the end result of: I can get a fully detected and labeled GRG in two minutes. But they're just outputs. And this freed up our engineers to really focus on the outcomes and what matters. Like — what's a building, anyways? How to handle any imagery on any phone in the field, and be able to fix it fast when it goes wrong.

And again, we did this in six weeks, and that's fast. But fast only counts when it holds up. Because when a GRG is wrong, a team ends up at the wrong door.

So bottom line: it has to work. It has to work well. But also just as importantly, it has to be in their hands now — not after another testing cycle, not delayed for more quality assurance tests.

So what we did — and I'm so glad JD went before me because I get to say everything JD said — we encoded architecture decision records as code, all the different PR/MR standardizations, testing benchmark frameworks. We encoded from day one to set our quality bar high. So every single code commit, every feature slice — doesn't matter — met the same quality expectations the whole way through.

I hear a lot of "AI ships spaghetti." That's not true. Developers do — bad developers do. And everyone with AI is a developer now. So the point is: you have to set the quality standards up front.

And finally, back to the question that was handed to us — "Hey, can you do this?" Short answer is yes. Buildings detected automatically on any imagery. Labels operators can actually follow. End-to-end GRG workflows on both Android and Windows. Fully offline on the devices they already carry. Security standards and operational testing requirements met — all in six weeks. And it's in use.

Okay, what does that mean to you?

Here's what that means for the mission. A large urban GRG — such as Charlotte — could take over eight hours. Now a complete one is ready in a few minutes. Smaller ones save about 30 minutes on average, we tested. And just as important: no more cut-short GRGs. When crucial detail gets missed because the operators have to go before they're finished labeling the GRG, it increases the risk to everybody on the ground. Now they get the complete one. All of that time goes back to preparing for the mission.

Let's go back to where we started. A pilot on the ground and the clock running. The value here was never the AI. The value is the minutes handed back to the operator preparing for the mission.

And because AI handled the mechanics, that same team of operators can now handle the next mission.

Speed to field is a requirement, please. It is not a nice-to-have.

Kenny, back to you. Finish this up.

---

Kenny Slater

If I said one thing to this room — start from the mission, not the tech. Our North Star was how quickly can an operator get a GRG, and how to best improve their process along the way.

Once you've done that too, hand AI all the mechanics, and you can focus your people on the mission. Define the outcomes, the value streams, improve those. That's your real engineering targets.

Three: own the quality bar. Set your ADRs, your specifications, your quality standards up front, and once you have those established, you can move fast and have stability throughout.

Four: do more with less. This was six people, six weeks, just trying to figure everything out as we went. And you get eight or nine six-week periods a year. So this is one snapshot. We've been doing this since, on different missions. And as they get used to doing this, I think time-boxing is really important as well, as other folks have mentioned.

And lastly: don't be limited. Get creative with AI. We had the limitations that we embraced with S20 and S23 phones on Android 13. No TPUs, very limited GPU acceleration support, Windows framework — all of this, we embraced it, and we used AI to get really creative on how to maximize the capability those devices have, because we can ship impact today with value to users right now, not waiting for the next generation of hardware to come in. So get creative.

And as Gene said, this was one of the hardest stretches I've ever worked in my life. But I'm also more excited than ever now about the future, because we finally get to build or fix all the little things we always wished we'd had.

So my asks from the room — what do you need help with?

One: it seems like a common theme is that human time is becoming a premium, a new constraint and bottleneck in these agentic workflows. How do I work-life balance now? It's a personal ask.

Second: more, better, faster models on government systems. This is my personal opinion, but I think delaying frontier capability on government systems is becoming a security risk. Again, personal opinion, not an official statement by the government by any means.

And then lastly, I just want to say: with AI, I can engineer a better world, and I love it, and hopefully save some lives in the process. That's what this work is for — making the moments count for those who need it most. Just want to leave you with: maximize the moment. Thank you so much.

---

Thank you, Kevan. Thank you, Kenny. Thank you for that amazing experience report.