Lightning Talk: Find your Fixer!

We have talked to hundreds of companies about how they find the right people in their organization to fix urgent problems. We asked them about their production incidents, their broken deployments, their vulnerabilities, major team changes, buttons that don't always work.

We ask them about how do they find their problems, and where are they in the environment? Why do their problems matter, and how do you find that right person? How do you find that original developer, the system owner, the sysadmin?

We've all heard stories about these terrible, terrible things that have happened in companies. We've walked through the aftermath of the events with their teams. We hoped to uncover some sort of new secret strategy, a new method that cuts through all the bureaucracy and red tape. We looked for that silver bullet, and I think we all know it doesn't exist, right?

We ask folks, "How do you find someone to fix that problem?"

And they say, "Well, yeah, we do what we do. It's not that bad."

So we ask them really seriously, "How do you do it? Do you dig through your CMDB? Do you use your service catalog? Do you have inventories?"

They always laugh when we ask about their documentation, and they say things like, "Is that still a thing? We don't actually have one. It's never complete. The data's drifting. It's always bad."

And that's when it happens. We start to hear phrases: "It's always been done this way. Nothing can be done. We can't change this place."

And my personal favorite, "It is what it is."

It turns out finding someone who can fix a problem in your organization, the right person, isn't actually easy.

We hear stories of 70-person conference bridges with nearly every team in the company involved. "We swarm on teams all over the problem," they say. "We're just going to keep paging folks until we find someone who can solve this problem. We have to appoint an arbiter on these calls because there's just so damn many people that we have to keep track of who can talk and when."

One company we talked to, they had all-weekend-long marathons with every single development team and ops team involved just to find out if the company was involved at all or exposed in the Log4j, Log4Shell vulnerabilities. This wasn't to fix anything. This was just to find the problems.

We hear stories of entire dev teams being let go. James hooked me up with that one. And production systems being left to rot with nobody to support them.

We hear stories of massive spreadsheets from hell, chock-full of tons of vulnerabilities being sent to engineering leadership, just waiting for somebody, anyone, to raise their hand.

And the icing on the cake: we hear stories of sending someone to drive to an employee's house to track down that one person they thought could solve the problem, only to learn from their spouse that they were at their kids' baseball game. You can guess what happened next. And newsflash, he wasn't the right one to fix the problem.

We hear stories of lost holidays, 2:00 AM wake-ups, disrupted family plans, and overwhelmingly we hear, "Well, we just keep calling people. Again, we just keep calling people. And again, we just keep calling people."

My name's Brian Walter. I'm the CEO and Co-founder of OpenContext. I've been through all of these types of things in my career: the late-night SWAT bridge, the after-action reports, the team swarming on problems. I've been paged, I've been called, I've been woken up at every hour of the night.

And the problem is I've done that to my teams. I have done all of this, and I'm just as guilty as probably the rest of us in this room.

The human cost of operating our environments is absolutely enormous. We seldom give it enough thought. There simply isn't enough time here to share all the horror stories I've heard, but what I can tell you, as leaders, I believe we're failing our people.

We have to understand that our companies and our systems are not just a bunch of code, not just a bunch of infrastructure, and not a cloud provider, but rather we're a complex socio-technical system with people at the very center.

I'm here to let you know that you can do a better job. We have to respect our employees' boundaries, and with all the great observability platforms and dependency mapping tools and context maps, all of these things can be improved. We can improve people's lives.

I was there in the room at Velocity in 2009 when this came up. I love this chart. This talk is a plea, a call to action, if you will. Embrace our people. Embrace some empathy.

And I'll leave you with this one. I'm Brian with OpenContext, and in the wise words of Bill and Ted, be excellent to each other.

Thank you.