From Fire Drill to Operating Model
Dustin Warner, director of software engineering at NRC Health, tells how a problem with no emergency became the first gear in changing how the company works. He describes pulling together the most AI-conscious people, moving past standard project planning, breaking teams into groups of two or three across technology, sales, operations and service, moving WIP limits to the new cross-functional teams, and supporting a growing community of citizen developers.
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
Okay. The next speaker is someone whose achievements I admire so much, and I've actually been able to follow his work so closely over the last year. His name is Dustin Warner. He is Director of Software Engineering at NRC Health, which was founded in 1981.
And so if you're in the US and in a hospital, there's a good chance that you'll get a survey afterwards, and that it will come from NRC. So NRC is who helps hospitals and health systems hear what their patients are actually experiencing so that they can act on it. And so they survey over 35 million patients a year, and more than 260 of the top 400 US health systems are their clients.
And I am aware, for those of you who live here in the UK, you might find this all very confusing. We have a very confusing healthcare system in the United States.
Anyway, the first time I met Dustin, it was just an amazing experience. He's a second-line leader, but he had Visual Studio Code up. We were doing massive SQL queries, and I've never seen a guy have so much fun coding.
And so in San Jose in April, he told us about how their platform, which one of their flagship products was reliant upon, was actually bought by a competitor, and how he built a very tiny team working AI-first. They were able to rebuild it in 90 days.
And then he ran into the next problem, which is actually how to get the rest of the organization to re-platform onto it.
And one thing I'll just mention about Dustin is that what I find so interesting about him is that he is doing technical practices and management practices that I have just never seen before. For example, he has a team of developers in Vietnam, and whenever he sees them asking a question, he will answer at midnight in Slack in Vietnamese. And so he just flew here from a hackathon that his team ran in Vietnam.
So, here to share his amazing story, Dustin Warner.
Dustin Warner
Thanks, Gene. So if anybody was in Charlotte, I'll be presenting a very similar but slightly different presentation here, titled “From Fire Drill to Operating Model.”
A little bit more about NRC Health. So we've been around in the US in healthcare for 45 years. We have over 10,000 partners at the hospital system level, so not just the individual hospitals, but the parent companies that run those. We've won Best in KLAS in 2026. So that is an award for technology in healthcare.
And NRC's real goal is to introduce human understanding in the full patient experience and journey in healthcare. So that's everything from how you decide where you go to get your care, what happens while you're there with our patient experience, what happens when you go to pay in the parking garage, to go pay your bill with our consumer experience. And then how does that integrate with the employees at the hospitals?
So if an employee's having a good day, does that mean a patient's having a good day? If they're having a bad day, how many patients have a bad day because of that experience?
A little bit about myself. I'm a director of engineering. We have three different directors at NRC Health. I manage about 30 engineers and three managers, but the real value here now is I want to throw in about 300 to 600 agents that I manage every week vicariously through my teams.
So of those 30 engineers, 10 of them are the top performers in the company with the most spend, but also the most spread in their AI capabilities and usage.
So as I read this coming to Bletchley, I realized “emergency” might sound a little disingenuous. We had a bit of a problem. There was no emergency. Nobody was dying. But we could lose some money very quickly. And what this turned into was going from an emergency to a catalyst that really changed how we went from a project to our product, and now how the organization is changing.
This was really kicked off and inspired by David Burick. Yesterday, Mallory was saying, “Find your Emily in an organization,” the leader that will attach to you, help make everything possible, and really be your advocate at the executive level.
And for me, that was David. David came into NRC last year, over 30 years of experience in healthcare in the US, bringing transformation to organizations, and really been an advocate for leaders that are pushing for change and making that happen for them.
So you'll see David a few times in the presentation today. I asked him to co-present with me. He wasn't able to make it, but we were able to record some videos and questions we had for him.
So one of the questions I had for David: We've had this team. We've had the tools and technology. We did not implement or use them until we had, again, not an emergency, a bit of a problem.
David Burick
We had an exogenous event occur which said to the top people in the organization, “We have to make change.” We did not have a grassroots need that was identified. It was an exogenous factor. It was like Y2K all over again.
And so what we're doing in this process, people just accepted the IT they got, and the creativity was in the workaround, not in the solution. And so we're bringing Dustin and his team to the table, and all of a sudden, as a larger expanded team, IT is inside of everything we do. That's still an aspirational statement, but I think we're getting there pretty quick.
Dustin Warner
All right, so Gene gave a little bit of history on our problem. We had a platform that was part of our flagship products, supported three of them, and it was purchased by a competitor. We had been on this platform for 18 years.
IT had almost no involvement in the implementation, management, or maintenance of this platform. It was all handled by the operations team, and that goes to David's point of the creativity went into the workaround versus the solution.
What happened was we got a clear mission. Renewal was coming up. We weren't going to be able to sign. Our CEO, Trent, set the expectation that we're replacing the platform.
After we've got the mission, we've pulled together the team. This may seem a little bit backwards. We didn't have our requirements. We didn't have our BRDs. We didn't have any of that ready. We pulled together the team of the most AI-conscious and forward engineers.
It wasn't about who's been using it the most, but who was the most curious and the most willing to try new things and also jump on a project that had a pretty high likelihood of failure, if I'm being honest.
So we got the team together. We flew on site and started working directly with that operations team, standing over their shoulder, having them demo, work with what they do, and then we started pulling apart the platform that we were having to replace.
The objective was 90 days. That was clear. We worked backwards from our renewal to being able to field these surveys in the new platform side by side to validate data, make sure there were no inconsistencies, and that we weren't going to be misleading our clients when we made this switch.
And the biggest surprise to me was we actually did it. We completed it in 90 days, and it was amazing. The excitement in the team going from our standard project management and planning on features and functions coming out in our platform to a team that we just dropped in and we're like, “Hey guys, we have to figure this out.”
I would've expected a lot of fear, anxiety, and stress. It actually came out with a lot of excitement, ownership, and teams trying new things that were unheard of at our company.
So the next big surprise for me was we're done with the project. The emergency or problem is over. What do we do now? We go back to our lives. We get on our projects, sit back, slowly roll into it.
But that wasn't the case. What we're seeing is with the AI, we were able to implement something that was thought to be impossible. We were quoted two years to replace the platform. We were quoted a year to get onto another vendor and still be in the same situation in a year.
And when the leadership team saw that, it was really the catalyst and the first gear that started turning the way we changed our organization. So we went from a project that had timelines, deadlines, no ability to move, to now we're moving this into our product organization.
So each project or product is now working in this capacity, and we're starting to see this change throughout the organization. So it was something that I thought was isolated to IT and the technology teams, but has actually started coming over to our sales, service, operations, in every aspect of NRC.
So we presented at the April Enterprise AI Summit, and I talked to Gene a few weeks later, and he asked, “How do you feel? This was great. Loved the talk. It was amazing.”
And I came back from that so inspired by this community and this leadership group of seeing what was possible and where everybody was on this journey. And my reference to Gene was I felt like this little guy, an angry dog ready to go, but on a very short leash.
And this wasn't due to the technology teams. The engineers were excited to start running to the next one. Really, the leash was 45 years of process and the way we've always done things.
So this is where David got introduced into the product line that was reshaped by this emergency, and we flew on site. He didn't pull together the executives that were supporting this. He pulled together the operational leads from sales, service, operations, and product and technology.
We sat together for a week talking about what the product needed, what were the issues, what were the gaps. We brought in so much information from our sales force on customer issues and complaints, brought in the data from sales, why they didn't win a lead, why they weren't able to close a deal.
And within a week, at the end—and David prints everything out, so it's fantastic. Emails on paper. He pulls out this binder out of his bag and throws it on the table.
What I didn't know is we had paid outside consultants to come in and work with our executive leadership team to put together a plan for this. And after seven months, they had a binder with no action items, no team, and no movement.
So in a little bit more colorful language, he looked at our team and was like, “This is not working. You now own this problem.” And it wasn't a technology problem. It wasn't a sales problem. It was all of us.
David Burick
What we have instituted is we've created a synthetic org chart. We haven't changed the org chart. That takes too much time. That takes political currency. It's a distraction.
But we've worked with principal people, and we've done some work that, although the org chart changes, people who had multiple products on their desks now have one product on their desk.
And so now that they have one product on their desk, we've created this synthetic community of people who are actually interested in the success of the project, of the product. Not just the sales, not just the customer service, not just the technology of it, all of it.
Dustin Warner
So we followed the same framework we did for our problem. And again, I'm really trying not to say “emergency.” We brought in for our little problem, we brought that same framework to now how do we apply this to our product?
We set the mission. We had to deliver so many features and capabilities within a set timeframe, and then, not able to share, but we had some multiple of revenue that we were expected to drive by the end of this year.
We built the team. Again, this wasn't a volunteer effort. It was, “You, you, you, come on, we're going,” and we all met up in Lincoln, Nebraska, our headquarters, and sat down to build out this organization and what it would look like.
And we set five objectives across those different teams, and it wasn't that sales was only in charge of sales, service was only in charge of retention, but it was everybody in that organization, the four team members and David, were responsible for all five.
And so we're negotiating, figuring out how we can help each other. I'm lending engineers to the customer service team to build out reports and build things quickly so that then they can come back, but give them room to close more cases. We're doing the same with sales now, of engineering coming in, building prototypes, demos, and getting things ready for their conferences.
The one question that I asked myself was, how did we get here? How did we get the right people in the room? And it really came from the very top.
Our CEO set the priority. It wasn't a technology project. It wasn't a sales project. It was an organizational project. He then put product line ownership in with David. David managed all of us, so now we all had two bosses: our organizational structure and David. And David is definitely the harsher of the two when pushing for outcomes.
We got the stakeholders and technologists together in a room. It was no longer build out the requirements and pass it over the fence. Even though we had a very strong internal technology org, we were treated almost like contractors in that there were only certain points you could pass data through, and those were the project managers and product owners.
We had on-site collaboration, so we meet once a month on site with that team and push for the changes and renegotiate what's the priority for those quarters. And then a big piece of this was assumptions started changing, and ownership over handoffs.
So all of our meetings are now recorded. We're using AI to summarize, bring our task lists, but we're also sharing those same recordings and summaries down to the engineering team so that they understand the mission across the organization and can start building with that in mind versus just the one feature that they're focusing on as a team.
So this was our org. We had our SVP of sales, VP of customer strategy, our director of operations, and that guy over there, the director of product and technology. We still reported all through our chain of commands, but now we had David as our leader, frontline, driving us for results.
So what did each group bring? I touched this a little bit, but sales brought the pipeline. What would close the deal? What were the deals they lost? What didn't we get because we were missing a feature or functionality?
Service brought in the daily customer touch points, and this was something that was amazing because there were points that came through customer service that would have never been prioritized. But with AI, it ended up being two hours of work, and we were able to take years of complaints off of the platform. And it cost almost nothing, but it never got prioritized because they didn't see it high enough in value.
Operations was the team that I've worked the most closely with, and this was interesting, discovering our real workflow. If you think as a technology team you know how your business operates, meet the operations teams that are supporting sales, service, and the customers and find out how many spreadsheets, homegrown applications, and sticky notes are running your business. It will terrify you.
I think one of my goals was to get VB out of our applications. We are now seven out of eight applications gone. The last one was deployed this week, and hopefully being deprecated by the time I get home.
And then product and technology, our role changed, too. We weren't just this outsourced company. We weren't this group behind a wall that we had to speak through the oracles of product to get to. We were in the room, and that was a change.
My managers are now coming into business meetings and being pulled in, and they were overwhelmed. But the ownership they got, the excitement they started pulling in—there's nothing better than the picture you showed, Gene, of Matt pointing at the screen like, “Yes!” We get that almost every week now when we're demoing and bringing things into production.
And my favorite line is, “When is this ready?” And we just say, “It's in production. We'll just turn it on for you now.” Nothing more exciting than the eyes light up because they had to spend hours and hours in operations to make this work, and now it's minutes.
So things that came with us from that emergency on the engineering side. One was working software out of every meeting, and that software would either be a feature function or an operational tool that was no longer needed, or we were able to pull closer to the product. So we would meet, the team would separate, go build out a prototype, and get it ready for deployment quickly.
Five teams of three. So we had much larger squads of seven or eight. We broke those to teams of three, maybe teams of two. I don't know. We're figuring it out like everybody else. My thought came with two people is an argument; three people, you can have a quorum.
And then build three and throw away two. This was one of my favorite stories of the project, was we had this design. We were debating as an engineering team, “Oh, we think this way will be better. We think this would be better. We think this would be better.”
And the ask I brought to the team was, “Well, how long would it take to write all three?” They're like, “We could probably get it done in two or three hours.”
Why are we arguing? We spent more time in an hour arguing about what we thought would be better. Once we built it, we merged two of them, threw the other away, and were able to move faster.
So like Bharti had mentioned, we've gotten a lot more ambitious with our experiments that we're running. We are willing to throw out a $5 bet on a bad idea all day long because we're no longer going all in on one of the concepts.
Our leadership has changed as well, not just in technology, but across the organization with sales, operations, and service. We're getting a lot more collaboration.
We've brought the teams together, so our full customer service team and sales team has now met the full engineering team, and they're putting faces to names, to requests, and really building up those relationships so that when we have something built, we're asking our partners to be testing that the same day.
And we've also changed a lot of our decision-making to the individual managers and individual contributors to make a call. If it's a two-way door, walk through it every day because you can walk back now. And again, it's such a cheap bet. We want to keep encouraging that.
The last thing I'll say on what changed in our leadership as well was we followed DORA, we followed Lean, so we've managed our work in process for our teams all day long, but that never helped us once we were getting things out into production, and they never got launched.
So we've actually switched our WIP limits from technology teams to that synthetic organization team. So no more than three things can be in the queue and being worked on for one request until somebody's signed off on one that's already been completed.
This has really put pressure on each other. So if sales wants a new feature, but service hasn't put in the time to test, we're going in that meeting, and the pressure's being applied, and I just get to sit back and smile.
What's changed with my teams? Managers lead the way. They're in the code. They're with the team, and the managers are either coming from a product or technology background.
But what's most important to me is if they can't demo the product and explain it, they really can't lead it. And we've had to move people around because of that, where they're wanting to manage people and not a product, and really leaning into, no, we're all in this together and having to work on these features.
Scheduled collaboration. So each team opens up office hours. We thought that this would be one or two people joining. We had 10 people join in our last office hours. We do them every Friday and Thursday, depending on the team, and they're the most engaging meetings that I've seen at NRC in my three-year tenure there.
Ideas are coming in. “This is great. We would love that. We want this.” Again, AI captures all of that. We feed that into our backlog, and then that synthetic org team meets once a week to prioritize all of this new feedback.
It's no longer days or weeks of email threads for feature requests. It's coming in, being reviewed every week, and then looking at feasibility.
Our next level on this, I'm working with my managers to say, “How do we take this from going to a team to going to the agents?” And we've put together a lot of frameworks now for taking these low-level requirements.
We have a product that has been around for a while, and they want small changes. Small, low-risk changes we're handing off to the agents from requirements all the way to pull request, doing a review, getting it deployed, and then via feature flags, enabling it one by one and safely rolling it out to our customers.
So we're ordering that feedback. We're taking it in. We're no longer saying, “Oh, no, you have to go formally request this.” We're listening to their voice, taking those meeting notes, and we're building those requests for them.
And then lastly, for my managers, leaning into being context amplifiers. I was talking yesterday with Matthew about before, cognitive load was seen like all the information you need to know could overwhelm the engineers. So we put in so many layers in our organization of, well, we'll cut it down by having a business system analyst on the business side. Well, they'll cut it down taking it to the product. They'll cut it down taking it to design.
And by the time it gets to the engineer themselves, all they get is this one line, “We need X,” with no idea of how it feeds into the larger system.
With AI, what we've been able to do is now hand the full context to the engineers, allowing them to ask questions of, where did this request come from? What meeting was this in? What was the sense of urgency around it? And really change how we're prioritizing and what the engineers get to see.
Because if they have ownership, what I'm seeing is the quality of code, the time for them to spend with the business is going up quite a bit because they own that product. They understand why they're building it, not just a ticket in a queue.
So lastly, what I'm looking for, what I need help with. Our synthetic org is going great right now, but it's not going to scale against every product. So how do we reorganize not just the technology teams, but really bring Team Topologies to an organizational level and understand how, in AI, we're turning new ideas into products daily, not every quarter or every year on a release?
So if anybody's really leaning into bringing engineering and the organization closer together, I would love to talk and hear how that's going.
The other, Michael mentioned this, and I heard it just a few minutes ago from Gus at Morgan Stanley. We have citizen developers now. They're building everything. It's amazing.
I am the guy, I try and be a lot like Gene, to jump into the call and be like, “This is amazing,” without giving the little bit of cringe of, like, how many Python packages did you pull in? Where are you pulling these Node.js packages from? How many vulnerabilities do you have in this, right?
I'm trying to encourage them to keep the excitement, but really how do we move from that into a product that we can really use, especially on internal tools? Because we see those as low risk, but realistically, if it's now a fundamental risk to the business and part of our delivery pipeline, that's something we do need to have controls and operational excellence around.
And then lastly, I love Kent Beck at the April summit mentioned none of us know what we're doing. If somebody tells you that they do, they're lying—or they're a sociopath. None of us would know. We're all figuring this out together.
So I'd love to hear where people are in their journey with AI adoption, both internally with their organizations, but externally with the community. Because it's hard waters to navigate.
You get online, and you see, “This is dead. That is dead. This was never working. Look at my product.” Somebody else, “Oh, I rebuilt that product in a day.” It's a hard time to really try and figure out where you should be going.
So I love connecting with this community. It's fantastic. Gene, thank you so much.
One membership. The IT Revolution library.
IT Revolution's books, the video library, the courses, and the MCP. The papers stay free.
