Speeding to Resolution with Human-in-the-Loop Automation

Hi, I'm Tina Huang, founder and CTO of Transposit.

But before I started Transposit, I was a software engineer where I had to be on call for lots of different production services. I started my career in 2001 at Apple, working on the application frameworks team, following my first true passion, building for developers. I was then fortunate enough to join Google in 2005, where I got to witness the rise of SRE. Back in those days, SRE was a really limited commodity, so you had to earn your SRE through runbook coverage. And then at Twitter, I built and ran a number of high-scale production services.

It's crazy how we all have these war wounds from being on call. I still remember one of the first times I was on call for Blogger. Back then, Blogger ran on Windows machines, and yet our development was all done on Macs. And so it wasn't until the first time that pager went off that I realized I didn't even know how to open up a Windows command line. And then another time I was on call, I checked to see that you could view the blogs, but I forgot to double-check that you could actually post updates. The next morning, I went into the office and my boss was livid. Blog posting was down all night long, and yet there was almost no documentation telling you how you were supposed to verify that the site was up. You were just expected to know. He pretty much brought me to tears.

All of these experiences, they really fueled my passion for building systems that reduce the need to rely purely on human intuition. At Transposit, we're attacking this problem, but doing it with a very, very holistic approach. At the core, we evangelize something very simple: how do you turn all of that institutional knowledge into codified processes? And the specific philosophy that we take is human-in-the-loop automation.

But first, I'd like to set the stage with what does it mean to have downtime in 2020? As we're all painfully aware, COVID has changed our lives in almost every way. As we bunkered down, businesses all had to turn digital for the first time. We basically took five years of digital transformation and compressed it into about six months. But as demand for online services boomed, so did incidents.

One sector hit hard was education. Students and teachers were struggling with how to start with remote learning, and they turned to Zoom in order to be their remote bridge to each other. But one Monday morning in August, Zoom went down, and it didn't come back online for almost four hours.

Almost all tech companies, including mine, have gone fully remote during the pandemic. This means we are more dependent than ever on platforms like Slack and Microsoft Teams. But in May, Slack stopped sending messages for almost three hours, destroying tons of productivity.

Then in August, for more than four hours, Gmail users couldn't send emails and many other apps in G Suite were also inaccessible. Only a month later, it went down again. And just a few days ago, Microsoft 365, including Outlook Mail, Word, Excel, and Teams, were hit with a massive outage. And the list goes on.

These services are more critical to our functioning than ever, causing higher demand, which is a good thing. But this increased usage often leads to more incidents. The next major outage isn't a question of if, but when.

Through the rest of this presentation, I'm going to walk through a bit of history of how agile and the rise of SRE methodologies have changed the landscape of what it means to have an incident. Then, the fundamental challenges of running a modern incident management process. We'll talk about some of the shortcomings of the traditional approach to automation, and finally, what I call data-driven human-in-the-loop approach.

So how did we get here? From the development side of things, the last decade has been all about agile and the shift to continuous integration and deployment. But as we are maturing in that practice, we are realizing that it brings with it new challenges on the operations and process side.

I was recently talking to an analyst at Gartner who commented how most companies are still riding the high of continuous integration, but they're about to feel this massive pain because they haven't yet created processes to make observability and reliability sustainable.

A lot of people focus on SRE as the answer, but really, SRE is a specific title. Whether you have an SRE org, you believe SRE is a role, or you just believe that reliability and resilience are important, what I'm about to describe is relevant to you.

I come from a software engineering background, so when I first approached reliability engineering, I really thought of it as engineers who just specialize in infrastructure. But then I stepped back and thought about what does it really take to be a good SRE? The first is lots of domain knowledge about infrastructure. But they also need to have lots of ability to automate through building tools. They need to understand what it takes to create good processes. But most importantly, they need great intuition. They need to have that full context of their production environments all in their head. And not just the production environments, but organizational context: who do I go to and in what circumstances? And they also need generally good intuition about risks and trade-offs.

And when we think about the impacts of agile, it's easy to focus on development and ops. But when we talk to customers, the problem is intricately tied to customers and customer support. When I was talking to my CEO, Divanny, about this a while back, I was reflecting on my experience at Apple. At Apple, we shipped boxed software. But what did that mean? It meant we had months between releases. That meant we had long bake periods for software hardening. We had feature freezes, UI freezes, code freezes, and strict change management towards the end.

And then Divanny pointed out something that I had never considered. The old service desk model was really built around support that was mostly about debugging user error. With these long cycles, there was no ability for engineering to roll out a change quickly. So if it wasn't a user error, you had to just figure out how to help the user work around their problems.

With agile, we are now testing code in prod. I don't care how many automated tests you have, customers are still uncovering bugs in your systems. Incidents often begin with customer support and then flow to engineering. Agile means that customer support is part of reliability.

Agile has fundamentally changed what incident management looks like. It has increased the frequency of incidents at the same time that it's broadened their impact, all at a time when service uptime is more critical than ever. Not a great combo.

So when we think about incident management, we need to focus on: how do I resolve my incidents faster, and how do I learn from and have my organization get better over time? I'm known for my controversial opinions, so here are two of them. One, a fully automated world is not the panacea we think it is. And two, postmortems alone aren't the answer to continuous learning when it comes to incidents.

So let's start with automation. 100%, we need more automation. And not just automation of technical tasks like cherry-picking a change or restarting a server, but automation of the basic incident process itself, like assigning a commander, filing Jiras, and starting Zoom bridges.

But most teams today are still relying on manual and chaotic processes. Teams that we've talked to overwhelmingly report that they have little to no automation around incident response. What automation they do have is fragile and creates risk.

So why is this? What I hear from our customers boils down to three things. First, automation is scary. Second, many processes have too much nuance to automate. And third, talking to all the services you need to automate is hard.

So let's go into each one of these one at a time. Automation is scary. Automation is scary because you need to have a lot of trust. You need to have full trust in this automated script that you may or may not understand, and that trepidation extends to how to run the script and the environment in which you're running it. For me, this pain point is always felt most acutely with Python scripts. I can never remember what version of Python I'm running or how to toggle between them. And then if something goes wrong when you run it, what do you do? How comfortable are you debugging it? It works until it doesn't.

Automation is about more than just the script. It's about the ecosystem where the script lives. It's about discovery. How do members of your team learn that this script exists? Documentation. How do you run the script? And maintenance. Who owns the script, and who can help troubleshoot it?

The second piece we hear from customers is that their process and systems have too much complexity and nuance to automate. Another way of putting it is you can't automate the unknown. If you look at the landscape of automation tooling we have today, you have things like GitHub Actions, AWS Lambdas, and Jira Workflows. All of these tools share one thing in common. They are trigger-based. Think about it. They all have some machine-detectable trigger that spawns a headless process. That means you need to have a reliable machine-detectable event. That headless process can't take any human input. If something goes wrong, you need a way to notify a person to step in. And debugging what went wrong is hard.

Engineers often like to say to me, "Hey, if Transposit is successful and helps me automate away all of my incidents, won't you be out of business?" I find this so adorably engineering-minded. Incidents are, unfortunately, a result of forward movement and innovation. As you introduce new systems, you will find that they interact in surprising ways, and those interactions can cascade into more incidents. Unless your product has been put into maintenance mode, you will have incidents.

And finally, automation is hard because APIs are hard. The connection between APIs and automation might not be obvious at first, but as our tool chain has moved to SaaS and the cloud, automation now requires coding against APIs. Here, I have a map of continuous delivery tools. Whenever we show this map to our customers, they typically laugh and say, "It looks so simple and organized compared to what we have." And this is just delivery.

Most of our tools these days are these powerful SaaS platforms, and that means automating around them requires APIs. Even our cloud infrastructure, like AWS, is accessed via APIs. And this brings us back to APIs are hard. Each one is a unique snowflake. Calling them needs to be secure. And finally, reliably scripting against them is complex.

This effectively means that to build automation in this world, developers need to have some general competency around distributed systems and security. For instance, they need to understand how to code against unreliable networks. They need their code to be resilient to long latencies of network requests. And they need to know how to write code that throttles requests appropriately to not overwhelm the underlying systems and trigger rate limits. To lower the barrier to automation, we need to make it easier to build on top of APIs.

So this brings us to we need to rethink automation. When we talk to engineering leaders, they often start with automation, but what they quickly get to is they want to take knowledge that is buried inside individuals' heads and turn that into a repeatable process anyone can do. When it comes to reliability, what we really care about isn't automation, it's codified processes. Codified machine process is automation, but a codified human process is runbooks. The ideal world is about being able to seamlessly bind these two together, something I call human-in-the-loop automation.

When you look at other mission-critical industries like flight pilots or medical doctors, they often employ checklists. Human-in-the-loop automation is basically checklists on steroids. Think of a checklist that documents the pieces that need to be done, but with buttons that let you easily run automated pieces. Keeping humans in the loop surprisingly means more automation and more robust automation. That's because you can let human judgment step in where the logic to automate would end up being so complex as to cause more problems than it solves. Where the logic would be so closely tied to the specifics of a fast-evolving product that it would need to be updated almost as frequently as the product itself.

So let's go back to one of our earlier observations. What makes a great SRE or reliability engineer? One, having lots of operational expertise. Two, lots of domain knowledge mapped in their heads. And three, great intuition around how to debug an incident when they occur.

Now I'm going to walk you through how we take that great intuition and turn it into a documented process that others can follow. How to make that context available to a wider range of engineers to enable them to use good judgment to make sound decisions. And then how to continuously collect new learnings to improve your reliability processes over time.

There's a customer that we're working with that talked about how oftentimes incidents are caused by these long-running queries in their Mongo database. There are a few problems with this. One, most engineers on the team don't know to look at MongoDB for runaway queries. Secondly, most engineers don't actually know how to kill a query. And third, most are afraid of killing a query on a production database.

What we worked with them on implementing is really simple. Start with a high-level basic incident management process. For us, we go to Transposit when something is wrong. The important thing here is you can only expect all on-call engineers to remember something this basic. Everything else should flow from there, whether it's filing a JIRA, starting a Zoom, or looking at a specific graph.

You can now organically introduce new processes, including driving engineers to better utilize runbooks. Then build runbooks to disseminate important information, like when you see this alert, check MongoDB. Finally, incorporate automated workflows in those runbooks to make it less intimidating and less error-prone to run tasks like killing MongoDB queries.

It's important to realize that a lot of what we need to do during an incident, we do infrequently. SREs have the advantage of focusing on reliability 100% of the time. But as one customer told me, their on-call rotation is every 24 days. So between tours of duty, they often forget what to do. Process is key.

So let's go back to my second controversial opinion. Postmortems alone aren't enough for continuous learning. Something that never stops surprising me is why incident management, which is so closely aligned with engineering, isn't more data-driven.

The other day, I was reading this article in The New Yorker about the Epic medical record system. When electronic medical records first came out, the promise was to have all this data available to help diagnose and treat patients at the time of hospital course. But in practice, the electronic medical records and the systems around them were limited to billing and accountability.

As I was reflecting on this anecdote, it occurred to me that incident management has exactly the same problems. We focus on root cause analysis as part of the postmortem process. Our ability to make use of that data is no better than with electronic medical records. We do postmortem analysis to understand what happened after the fact, and we sometimes file tickets to track areas of improvement. But more often than not, postmortems are used for documentation and accountability. That data is inaccessible to on-call engineers at the time of the next incident.

Almost all customers I've talked to use something like Jira or Google Docs for their postmortems, and all of them say that that information is not easily searchable during the next incident.

The way we think about incident management is fundamentally broken, and the data we collect around it is sparse or missing. Most incident management platforms treat incidents as these discrete events, and they track effectively the single metric MTTR. But this is broken for a few different reasons. Incidents don't start at the alert. It started when someone did something like commit a bad line of code or do a bad infrastructure change. Incidents don't end when the ticket is marked resolved. That's usually just indicating that the service has been restored. And then everything that happens during an incident is effectively treated as this big black box.

We need to be more data-driven. We need to collect more granular data on what happens at the time of an incident. Things like who is involved, what runbooks are people using, what actions are people taking. If we had this kind of data, we could start learning in a whole different way. We could answer questions like, "Where are we missing runbook coverage?" "What are common tasks that take up a lot of time that would most benefit from automation?" "Who is being pulled into every incident and is likely to hit burnout?"

We need to start thinking about incidents as part of an operations continuum, and we need to start capturing more data across the entire spectrum of human and machine events. Let's stop and think about where all the data around incidents currently lives. Most teams use a number of tools like Jira, Google, and Zendesk to manage communications and process. And with communication, there's the big elephant in the room, chat clients like Slack. I can't tell you how many times I hear customers complain about how they can't stop incident communications from moving to chat, and yet they have no way of capturing the data there. There's a number of system tools like APMs and continuous deployment that all live in their own silos. And then there's external communication through status pages, blog posts, and email.

But in the middle of an incident, the last thing that is on anyone's mind is capturing data. Today, we rely on manual capture done primarily when building postmortem timelines. And this process is far from ideal because it's manual and time-consuming. I can't think of an engineer who doesn't complain about having to write postmortems. It's also error-prone. It requires you to sift through chat logs and go back and do screen grabs. It's unstructured, so it's hard to access and use during the course of the next incident, and it relies on humans to analyze it and learn from it.

So how can we eliminate the burden of having to manually capture all of this data, especially messy human data, and capture it in a structured way? I began my career working in consumer. When I worked on Google News and Twitter, we instrumented our interfaces to automatically gather data and help us improve the service. So it occurred to me, let's do that for incident management.

As we discussed, we want a single funnel point for driving an incident response process. Now, I'm going to add to that, you probably want more than just a framework. You want an actual platform. And that's because this platform will allow you to automate and control how you capture data. In the ideal world, this platform is easy to use and makes on-call life easier.

We often use the slogan, "Beach Ops." The dream is to have an on-call process so easy that you can manage an incident from your phone while drinking a margarita on the beach. Having a platform that everyone wants to use that also captures the data your organization needs is a win-win. This means you can stop using a stick to beat process and data capture into your team.

But having the data is not enough. We need to make that data accessible to on-call engineers during the course of the next incident. There's basically two parts to make this happen: capturing the right data with the right structure. If your data is unstructured text such as chat archives, it's going to be hard to make use of this data. And then create usable tools to access this data. Luckily, once you've adopted an incident management platform that automates the structured data capture, you're halfway there.

So what are ways of making this data more accessible? I'll describe two of them. Again, taken from the consumer world, recommendations and search. We can use that data to recommend things like when a particular alert comes in, suggest people to invite to the incident, suggest runbooks that might help with investigating and resolving the issue, suggest workflows that are commonly used.

And now that we have the structured data, you can enable much more robust search that allows you to answer things like, "During the hour before the incident, what changes occurred in my environment?" "Last time this alert went off, what was the cause?" "When the last person ran this workflow, what parameters did they use?" And everyone loves machine learning. Luckily, once you've got structured data, you can employ all kinds of existing machine learning techniques to help further drive improvements.

We've talked to hundreds of customers about their challenges with driving resilience into their organization, and we've employed all of the principles that we've just discussed to help them turn all the institutional knowledge into codified processes. Look, this isn't meant to be a sales pitch. You can employ these techniques in a number of different ways. But I hope these examples from Transposit will help you to incorporate these principles into your organization.

We've built something we call an interactive runbook as our solution to human-in-the-loop automation. Here you can see what it looks like to have a runbook that codifies human processes around something like debugging an alert for high 500s. And you see how the runbook seamlessly integrates with the choice of workflows that spawn automated processes. But the on-call engineer is in control of choosing if and when to run them.

Here is an example of what I mean by capturing semi-structured data around human processes. This screenshot is of the automatic documentation that Transposit collects. You can see we've captured elements like what runbook an on-call engineer is using and what actions the engineer is taking. You can see that we suggest runbooks and actions to help with a given alert, but the on-call engineer can choose whether or not to follow these suggestions.

And what it means to make that data accessible during the course of an incident. We've built something we call knowledge streams that lets the on-call engineer do sophisticated searches over previously recorded data.

And finally, what it means to abstract away the complexities of APIs. Here you see the developer platform we've built at Transposit. We support SQL, Python, and JavaScript, but I think SQL is the best example of this abstraction. You can see I have a simple SQL statement so that the developer just expresses what data they'd like to fetch, but they don't need to think about all the complexities of APIs such as authentication and pagination.

In today's world, reliability and resilience are key. To achieve uptime, we need to move from institutional knowledge to codified processes, and to do this, we need to take a data-driven, human-in-the-loop approach to automation.

While many companies aim to solve technical problems with technology, at Transposit, we aim to solve human problems with technology. Thank you very much for watching. I hope I've opened your mind on your approach to resilience.

If you have any questions or want to talk, join me at Transposit's Slack Q&A and visit our booth for a demo, live chat, and cool prizes. We'll be hosting two special happy hour events. Check out our booth for details.