From Fire Drill to Operating Model

Host Intro (Gene Kim)

All right. Our first speaker for the morning is someone whose work I've admired so much, and I have followed his journey almost week by week.

Dustin Warner is the Director of Software Engineering at NRC Health, founded in 1981. So if you've ever gone to a hospital and gotten a patient survey, there's a good chance that it came from them.

NRC helps hospital and health systems hear what their patients are actually experiencing and allows them to act on it. They survey more than 35 million patients a year, and more than 260 of the top 400 health systems are clients.

So I met Dustin through his boss, Tim, and their CEO, Trent Green, who I've known for decades. We were at a basketball game — go Portland Trail Blazers, I have to say. And Trent mentioned to me that one of his desires was to link higher patient satisfaction scores to increased use of medical services. And so next thing I know, I meet Dustin, who has Visual Studio Code up. He's generating massive SQL queries, and I'm thinking, "Here's a guy who's a second-line manager, and he's having so much fun coding."

So in San Jose this spring, Dustin told us how the platform underneath one of their flagship products was bought by a competitor, and how a tiny team working AI-first was able to rebuild it in 90 days. Which raises a question that so many of you are about to face. When AI enables switched-on technology teams to move fast, it puts enormous pressure on the rest of the organization, and suddenly the bottleneck is in seemingly every part of the organization but technology.

And so I suspect that you're going to hear — or have experienced — some variant of Dustin's story, and I have no doubt that you'll be grateful for him sharing the story to help move the needle on one of his organization's most important goals.

Please welcome Dustin Warner.

Dustin Warner

Thanks, Gene.

As Gene said, my name's Dustin Warner. I'm a Director of Engineering at NRC Health. We are a healthcare analytics and research firm.

Today I'm going to present — we presented in San Jose the 90-day fire drill, what happened when a competitor bought one of the products that we use underneath. And then since then, what do we do now? What does the organization look like after they've seen our ability to go fast, deliver quickly, and replace some of these core platforms?

So a little bit about NRC Health. We've been around since 1981. We are 425 associates strong, and we've got partnerships with over 10,000 healthcare organizations here in the US. As Gene had mentioned, we were one of the pioneers in the surveying industry to send out text messages, first to do the automated phone calls, and start receiving that from patients as soon as they left the door. So we deliver analytics on the full customer journey in healthcare — that's everything from how do you decide where you go, how were you treated when you were in there, and what do you do afterwards, whether that's payment systems, medical records, things like that. So we partner with these organizations to really bring human understanding to their customer base.

So a little bit about the emergency that kind of kicked this off, got me introduced to Gene, and really set a new path for our organization.

We had a product that was supporting 30% to 40% of the revenue at our company. They didn't own the full stack, but they owned a core piece. And this story is very familiar from previous lives — we've got to get off of Oracle, we have to get off of a mainframe system, we want to get off of Salesforce, all of these things. And normally that was years of planning, multiple teams, trying to figure out where to put in the seams, where to cut things out.

We didn't have that time. We had one year before the renewal was up, and we didn't know if we could even renew with the competitor at that point. And we did not have engineering teams laying around or budgets to just go hire 20, 30 engineers.

So our approach was three people, 90 days, AI-first.

And I measure time now in the models that we were in — so that was Opus 4.5. We've made a few steps since then. We flew out to their headquarters, sat with the team, walked through their daily, and then really changed how NRC's engineering teams worked with the business. Instead of requirements going from product and project management to the teams, we sat in the room with them daily, turned around demos daily, asked for their feedback — and it was a success. We were at feature parity in 90 days.

We did have a lot of learnings from this. Like, when we're moving this fast, how do we have the business sign off? How do we have them test things out? We've been learning step by step and day by day with that.

Proud to say we are completely off of that platform. We've been migrated. We are now running all of our surveys through the new platform, and it's been a success.

So the next part is really the hard part. This came from our senior leadership team: how do we keep the urgency of the team without the emergency? And at that point, it wasn't just the one team — it was how do we get everybody to work this way?

So we went from taking a team of three engineers that were trying something new for the organization to how do we spread this to the 80 engineers that we have now and get everybody working in this fashion?

We started to roll out new teams — we're reorganizing teams, really. And we found out engineering wasn't going to be the part that stopped us anymore. We could run through these changes, get features out there. What we found is inventory and backlog when it came to turning on and releasing these features.

So as we were finishing up that survey platform, we were meeting with the COO weekly. They brought in a new executive leader, David, who was going to take not just that part of the platform but the entire product value stream and grow it. We had a goal of 4X that product stream in two years — very ambitious goals — and he's wanting to run. They formed an executive tiger team. They all got together, started looking at what we were doing, where we were at, and they were doing that while we were getting off of the platform.

We didn't get much progress. So he pulled in a team of directors, managers, and VPs, and for one week we sat in a room going through everything we had: inventory of the systems, inventory of the processes. Where do we have technical debt? Where do we have process debt? Where do we have opportunity? What wasn't closing in sales?

At the end of the week, he takes this giant binder — and David's a very unique individual, he prints everything off, so emails, PowerPoints, it's all printed off and brought into the meeting room with him — he takes this binder of the seven months that they had spent on planning how they were going to do this, throws it on the table, and in a little bit more explicit terms says, "This is all crap."

And we've started meeting with that group weekly now. What we've instituted is we've created a synthetic org chart. We haven't changed the org chart — that takes too much time, that takes political currency, it's a distraction. But what we've worked with principal people, and we've done some work so that although the org chart stays the same, people who had multiple products on their desks now have one product on their desk. And so now that they have one product on their desk, we've created this synthetic community of people who are actually interested in the success of that product — not just the sales, not just the customer service, not just the technology of it, all of it.

So the urgency that we brought with the engineering team — dropping everything, flying three engineers out to the headquarters, meeting with the team — he now brought that across the organization. He's one of our executives, and he reached out to the bosses of these four individuals, including me, and said, "We need these people for a week, and we're going to need them for a lot longer than that." And he brought that sense of urgency that we had at the platform up one level into the management and leadership layer.

So we had our — the product is Market Insight, so we have our MISO group, the synthetic org. And what we've done is brought in Brian, our SVP of Sales, only focusing on that product now. We brought in Ryan, our VP of Customer Service, only on that product now. Heather's the Director of Operations — she's only focusing on that product. And then what we didn't have was a separate column for product, a separate column for engineering, a separate column for PMO, a separate column for design. We brought all those just to this group of four, and we're meeting weekly to go over the priorities, go over the platform.

It's been amazing because I'm seeing our sales wins every week. I'm seeing our customer service losses, where they've had good meetings, where they've had bad — and that feedback is coming in real time.

We're using AI every step. So we're taking those meetings, recording them, doing outputs, sending those to the engineering managers. And what we're looking for is both what is the press release we want to have out there in three months, but also is there an issue that is now a four-hour change that an engineer can just pick up? So we're using AI — whereas before we had filters for context through different roles and positions, now we're using AI to get the context directly to the engineering teams, getting those turned around, and in many cases the day after this group meets, we've got product demos posted in Slack for the team to review and sign off on.

So what do we need from each group?

It's funny — in being in engineering for a while, we normally hate sales because they're selling things that we haven't built yet. We've kind of shifted that. Now we love it. I want to know what is the thing they're trying to sell in three months so that we can get it out there and in their hands in a month to start selling. So we really are looking for sales to bring us the pipeline and bring us the runway of what are the customers looking for. What is the one feature that would allow you to close that deal, and what is the visibility like? Every deal is important, but we all want the whale. So is there one feature, one product, one set of insights that we could bring to help close that?

Service — this was amazing. This is a partnership at NRC that had not really been there because of how we structured IT; it had been in more of a vertical silo framework. We got to hear weekly about every issue customers were having. And in that meeting, it wasn't just, "Oh, this person said this." It was, "Okay, log in, show it to us." We were able to grab those. And in those meetings, I'm sending off Slack messages to the team like, "This is broken. Let's jump on it now." And before, that would have been, "Oh, we'll talk about this in the huddle tomorrow. We'll get this on the next sprint." Now, many of my engineering managers, their first pass is just putting it through their AI agents. Is this something they can fix? They get a pull request out there. They look at the impact. If it's low impact, low risk, it's out the door — sometimes before we walk out of the room.

Operations was where my heart really hit, because this revealed the real workflow of how this product was running. There were so many spreadsheets, so many little knobs and dials that were being twisted every day to be able to get these products out the door, that were hidden behind these Herculean efforts of the operations team. So we spent a lot of time with them. We started introducing the idea of not just technical debt, but what is process debt? What is the thing we've been doing for 10 years and have never changed because we've never had the priority or the opportunity to change it?

And what's been amazing is so many of these things are two days. We take one engineer and two days and cut eight hours off of their week. We take one engineer and take something that they've absolutely hated and that introduced manual errors monthly, automate that into a system, and take that off the operations team. So operations is near and dear to my heart because we've been able to deliver so much joy to them in using the platform. And I wish I had a slide of all the feedback that the team's gotten — bringing them directly to that team and helping them out.

Product and technology — our role changed too. It was no longer, "Here's a spec. We need this. Here's everything you need to know about it. We've been working on this for two years. Go build it." Now it was decisions in days. In that meeting, I'm put on the spot. I have to know our systems, our abilities, and our capabilities because they're asking, "How long is this going to take?" We're terrible at estimating, but we do our best. And they're looking for decisions in days — just like if they pull up ChatGPT and ask it a question, it's not going to say, "Let me get back with you in two weeks." It's going to give you an answer. And if it's not confident, it'll let you know.

So we're bringing real-time decisions. We're bringing solutions in as well. So instead of waiting for problems to come to us, we've put in monitoring, observability, and we're seeing where friction points are and bringing those into that meeting. We've noticed when users come into our platform, they're spending a lot of time here — this is because of loading, this isn't because of an actual feature that they're utilizing. They're clicking in this area. It's had a "Coming soon" for three years. Maybe we should remove that.

So just throwing out a couple of quotes.

From Heather: "The speed, flexibility, and collaborative approach have fundamentally exceeded the expectation of what's possible for a project like this." For our platform, it was estimated as a two-year project by multiple firms we brought in.

From Ryan: "The AI-first approach has allowed us to quickly develop and test new features within days and create more efficiencies with the customer success team and turn out a better experience for our partners." This has been amazing because it's been not waiting for a meeting to get feedback. We set up dedicated channels — as a feature comes up, we create a channel, we're posting daily videos, we're getting feedback from the team in real time. When we're ready, we close that channel out, but we put it into onboarding, and then their team gives us daily feedback. And we prioritize it now based on the difficulty and the ability for AI to do the work. So if they say, "Hey, it would be great if we had this default, if we had this optionality" — we feed it through an agent, let it build it all the way through, we review at the PR level and say, "Okay, this is introducing risk. We're going to table this, bring an engineer in to work on this, and get it to the next level." Or if there's no risk — UI changes are one that we've got extensive test suites around, we don't have a lot of risk on data that's being exposed or anything like that — those ones are coming through almost daily of, "Oh, it'd be nice if this was over here," or if we adjusted how this user experience flowed.

So that was the core team, the MISO team. What we also have in those weekly meetings are our Chief Strategy Officer and David, our EVP, coming in, and they've set out five goals for us.

One of the goals that David brought in is: how will we use our IT investments to improve margin and growth? This was normally a question reserved for the executive team — they would work with sales and service, build out these plans, and then deliver them: "These are our priorities." Now it's coming into this group, and we're able to have these discussions. So this could be anything from cost savings on the technology side to identifying the one feature that would allow us to close the deal and move into growth.

Bringing in customer service and operations, one thing that's really stuck out is the attachment to spreadsheets. We loved our spreadsheets. We'd never get off of them because we could do that in a day, and it would take engineering six months to give us half of what we wanted and a lot more of what we didn't. So this is where we've really leaned into systems, not spreadsheets — moving the team away from that. And honestly, a lot of this has been allowing them to kind of go in, vibe-code with their spreadsheet, articulate what it should look like, our team taking that and being able to turn it into an actual product that gets pulled into the operational systems. So we've killed five Visual Basic apps, about 20 spreadsheets that all did about the same thing but for different people, and been able to give that same functionality across the team in a very tailored experience.

VB and Delphi is almost out of NRC. We're not there yet, but we're almost.

So when David came in, this was the question he asks every week in this meeting. We've got five goals — this is one of them, and he expects an answer. So we ask David, once we've agreed on these, what are the next things? And one thing that I always ask is, why didn't we do this before? Why did it take an emergency for us to realize our teams weren't structured right? Why did it take an emergency to realize that silos were bad, even if they were specialties within front-end, back-end, database engineers? Why did it take us so long?

So this was David's response on why it took an emergency to get us to this point:

> We had an exogenous event occur which said to the top people in the organization, "We have to make change." We did not have a grassroots need that was identified. It was an exogenous factor — it was like Y2K all over again. And so what we're doing in this process — people just accepted the IT they got, and the creativity was in the workaround, not in the solution. And so we're bringing Dustin and his team to the table, and all of a sudden, as a larger expanded team, IT is inside of everything we do. That's still an aspirational statement, but I think we're getting there pretty quick.

So one thing I loved — David was supposed to be here with me. We ended up doing an interview because some challenges came up. But one thing he brought up a couple of times was that creativity went into the workaround, not into the system. And not only have I seen that at NRC, but I've seen that at about every company I've been in. We applaud these Herculean efforts to get things done, but then we leave it there. And these people are bound to the solution that they had built and support, and we never take that off of their shoulders.

So when we were looking at what comes after the emergency — we met with the executive team, we reshuffled. All of the teams under me were broken up, reshuffled, getting the right mix — both technical aptitude and also ownership, like who's really going to dive into this.

But we moved. From emergency — driven by an external factor forcing us to do this — we now have our product goals set by that team, and we put just as much pressure on the teams with those goals as we did with the external factor.

For the emergency, we had the CEO and the COO of our company in the room about every week asking us on progress. Where is this at? Now we have that cross-functional team. I could text any of them right now with a question, and they would text me back within 10 minutes. They would drop things to get things moving.

How tech was seen — so since 1981, we've always treated tech as this external factor, these people over here. And we've now brought them into solutioning, brought them into the meetings with sales, service, and operations, and taken a lot of that pressure off of them to build the workarounds and started pulling us into the solutions.

So speed became the expectation. I'm going to go pretty quick through some of these slides. We had a SQL upgrade that had failed twice. The teams previously had estimated it at nine months to a year. We brought together another team of three with the expertise, with the tools, and with an environment set aside. Two and a half weeks — we had completed that upgrade, no data loss, no interruptions to applications, about a 30-minute downtime in the middle of the night when none of our users were online, so no impact. And that was a two-and-a-half-week upgrade versus the nine months that had failed twice.

We're turning around products that we were estimating at six months — features that had already been in our backlog for a while that the team had estimated. We've now taken those, brought them to a small team, and said, "Here's your token budget." That's a fun experience trying to figure out right now — I think everybody is. But here's your token budget, here are your teammates. Do you want to keep one and get more tokens? Do you want to lose one? We're letting the team decide, and we're turning those around in a few days. We're getting actual clickable prototypes in the user's hands within four hours.

And a big piece that I've really enjoyed is doing four-hour hackathons. This was inspired by Gene's countless stories of meeting with people and getting that aha moment — you've seen the picture of Gene like this with about five people when they finally got it. It's been great sitting with the business. And it's not like we're making these efficiencies in operations based on what's the longest time, but it's sitting with the individual and going through, "What's the most painful part of your day? What takes you out of flow? What is the thing you push off to the very end and then introduce risk of manual error because it's the last thing you're doing?" We're looking for those stories, not so much the spreadsheet "oh, well, we saved five hours here." No — we've now reduced manual errors and cut all of these things. And for an operations team, they're not always thinking about those manual errors being introduced. So those hackathons have been my favorite part of the week.

All right, coming up on time. But we have working software in meetings — so we're prototyping during and directly after the meetings, taking AI summaries to feed those into our pipeline, not passing it through five layers of product owner, product managers, engineering manager before the team gets it on their next two-week backlog. We've broken down teams into groups of three, and now the big part is we build three times and throw away two. So there's a lot less cost with making errors that we catch in development or in co-design versus what goes to production.

So a few things that have changed. What I ask of my managers — their jobs got a lot harder. They're no longer proxies for HR. They are forced to be in the room building with the teams. My ask to my engineering managers now is you have to be able to demo every feature that the team puts out and jump on a call at any time — so be ready.

Remove statuses. I have daily updates coming from my Slack bots telling me what all the teams are doing, summarizing the messages. I don't need that from a manager anymore.

And then leaning into what have we learned from these experiences.

AI is the expectation now. It's no longer who's using AI, it's how are you using AI — and just starting off discussions like that. We're using analytics in the business of doing business. Our operations teams have Codex and Claude Code — they're building these spreadsheets into systems. And then when we're ready, we take that into our systems and actually build it to enterprise grade when and if it's needed.

Let me jump to the last part — what we haven't got. I know this is one of Gene's big asks because it wasn't on the slide to start with. I'm trying to figure out how do we get this organization to really take off. Volunteers don't scale.

Product, technology, and design have all merged now. We're getting product people getting into engineering, design getting into engineering, engineering getting into product and design. So how do these roles combine? Which roles get replaced? And how do these grow so that it's not 20 handoffs, but it's one handoff between your head and the agent that you're working with?

Adoption is still the slow burn. We put in some inventory control systems for the engineering team — excited about that. I'd love to know how others are doing that as you're turning out more features. How are you getting adoption?

And then my last ask, which didn't make it on here — if anybody is deep into consumerism and marketing technologies, I would love to talk to you. We're trying to do what Gene's doing: understand across industries, not just healthcare, but how are other people viewing consumerism and how are they working with that data?

Thank you so much for your time today.