Las Vegas 2020

Outcomes Over Outputs: Measure What Matters to the Business

Traditional Lean-agile metrics like velocity and burndown have measured and reported on effort - either in story points, or hours or days; more often than not to justify work completed rather than value delivered. However, effort only measures outputs which can be useful at a team level. The business, on the other hand, cares about outcomes and impact. When it comes to reporting to the business on work across an end-to-end value stream, teams find it difficult to isolate the effort needed for the completion of value units delivered. Typically, the business wants to see value-adding work, rather than activities completed - such as revenue-generating work, revenue protection work or internal improvements.


In numerous discussions over the past years on making work visible and about flow metrics like Flow Load (and setting WIP limits) and more recently, Flow Distribution (all of which rely entirely on the number of value-adding work items rather than effort), one of the most common questions that surfaces is what about the size of the work item? How does that factor in the reporting?


In this talk, Laksh Ranganathan explores some of the reasons why quantity and context rather than effort is a better gauge of value, and how focusing on quantity can enable teams to not just align better within the business context, but also better manage their capacity. This change isn’t easy and prone to pushbacks - Laksh delves into popular pushbacks to using quantity and how IT teams can address them.


This session is presented by Tasktop.

LR

Laksh Ranganathan

Senior Manager, Product Operations, Tasktop

Transcript

00:00:14

Hello. It's an absolute pleasure to be here today at the workshop DevOps enterprise summit at Las Vegas. I hope you're having a lovely time so far. I am Lux rundown. I'm senior manager of product operations at Tasktop. Um, and it's, it's, uh, I'm thrilled to be here and to talk about how we measure the right things, uh, to align to the business. I want to start with this quote, which is a constant reminder for me that just because you can measure something doesn't necessarily mean it's going to provide the answers that you need. Um, it goes on my whiteboard quite a lot to, to remind me to focus on the right things, really. So how do you determine what you should be measuring? Let's take some here, for example, um, this chart gives you a sense for the miles that he has been walking over the past few weeks.

00:01:15

How did the credit and tell me, what do you think? How was she doing? No matter what your answer is? Uh, I suspect it's likely true because it really depends if she's oh, to improve her fitness, she's doing fabulous. If her goal was to run a marathon next week, though, she has done the right amount of miles. She has been walking versus what he needs is to be running. Um, perhaps not as great. What if she was trying to prepare to climb a mountain, like Kilimanjaro, that chart, that doesn't quite provide all the information that you need? Like what sort of elevation is she doing, which is necessary if you want to be preparing to climb a mountain? Um, what about, um, the weight she's carrying? Is she carrying a pack while she is preparing? And it's, it gives you a sense she would have to consider those in order to be able to determine if she's really ready for that.

00:02:20

This sort of goes to the crux of the disconnect between it and business, because they have a different view of the word. So in terms of being able to determine what's the right things to measure, you have to first look at what is the goal, what are the questions that the business needs answering organizations are hierarchy for a reason as the number of people increase, you need that kind of a structure to be able to scale the operations. A consequence of that is that organizations, as they grow, become optimized for teams and functions, which is not wrong, you tend to have different departments for product management and development and testing. You have vendor dependencies, um, operations and so forth. Um, sometimes you do have, um, sort of, um, full stack teams, but that does not quite encompass every one of these operations. It's just not practical sometimes.

00:03:34

So when it comes to providing metrics aligned to business, um, you have to put together, uh, metrics from each of these functions to sort of paint that full picture. And what ends up happening is some of the, each of these metrics though accurate, um, are optimized for their own context, which means that they don't quite tell the story. Um, from that, at that business level, business predict, typically looks at things like how do we satisfy our customer demands, especially when it's changing, how do we adjust and pivot to that? Um, what sort of investments do we need to put in to provide the most value to our customers? How quickly are we delivering value? What is that rate? Um, notice that a lot of this focus is on the customer itself. So the question that comes up often is how do you measure this value from that customer B?

00:04:45

Um, it's hard and it's difficult to articulate because it varies depending on who you ask for it. Um, so the thing to remember is that value, um, when it comes to a customer transcends across all of these functions. Um, so in order to be able to track delivery, you have to look horizontally across the end to end. You have to put that customer lens and view it from that customer perspective, from request to release. And you can only do that if you apply a product lens because a functional, um, clerk or a project on click only looks at, at part of the equation, and doesn't quite, um, provide that customer perspective. You, you can only do that with a product or a value stream, um, lens, um, any place when we talk about a product value stream, that by definition, it has a customer identified and it produces a product.

00:05:56

And because of that, it tends you tend to be able to describe value because you know who the customer is and what that product is that they, um, consume from that particular value stream. And this is key because this also provides you clues on how to sort of measure that value through business results, which are associated with them either BA revenues, um, renewables, cost metrics, quality metrics, whatever that is. Um, but you would be able to relate that back to the product and to the customer. When you think about products within it, the one that comes to mind, our business products, products that have an interface with your external customer, but not all products have to be external facing. In fact, a significant portion of products within an it organization are internal facing. They serve internal customers, be it a business product, um, a internal team, um, or, or even employees to support them, to manage their work.

00:07:11

Typically supporting products could be a payment processor, for example, that, um, that support all of your business facing products like the point of sale system, um, or it could be standalone products like HR system, a billing system, a contracting system, which is absolutely necessary and, and waiters for employees to be able to do their job. And it's still within the purview of the it organization to manage. And the third layer there is the platform products, tools, and technologies, um, and, and, and basically any platform that is needed to develop each of those other, uh, internal or the external products. Um, typical customers for platform products would be developers and specialists, um, who drive or who derive value from that. And typically examples could be your IIS, your, um, CIC pipeline, your tools that developers use that need maintained. Um, if you don't focus on some of those platform tools, it can directly impact how quickly you can drive value through the other products or push value to the customers of the other products to determine how you would, um, you determine and chart of the products within your it organization.

00:08:46

The easiest thing to do would be to pick applications and start asking questions like who is using this product, what is the purpose of it? Who is deriving value, who is driving prioritization for these, um, who has a influence on value creation or value protection, like most offices, um, and, and who is paying and supporting this product, asking some of these questions will help identify the customers, um, and the influencers for this particular product. And in turn, it'll determine the right business outcomes for each of those products, which in turn can then give you a sense for the impact that it has on the customers. And that could be, uh, the results that the business desires out of that particular product. Once you get off point where you have some of your products, identify it, then becomes much easier to now start identifying what metrics will, um, will relate, uh, and drive that value or, um, for drive that business value for that product.

00:10:07

Now there's no shortage of metrics within software delivery. Most dashboards look like planes, cockpit controls, um, with way too many, um, measures in place. Let's look at this view. Here are both are a view of a bank statement. One a summary view. The other is a detailed view, which one do you prefer for me? If I was looking to find out how much balance I have before I made that big payment. Um, and I just want to make sure that I have enough funds to cover that first one does the trick. However, if my focus is to cut non-essential expenses, then I would be looking at the second one to figure out what changes I need to make. So the metrics for the business are no different. You just need to be able to align the right level of detail, um, and align them to the context.

00:11:13

Um, and this is where sort of flow metrics come in. Your standard software delivery metrics are great and are absolutely necessary because they provide vital information about, um, team level, um, team level functions like processes in level decisions like capacity planning, um, utilization, optimization standards, and so forth. However, they tend to have too much detail to be able to, um, to, to be able to easily align to business, excuse me, business results. And this is where flow metrics come in, which provide a more aggregated view. All of those metrics in such a way that it's correlated or which can be correlated to business outcomes easily. The two are two sides of the same coin. You can't have one without the other, and you should be, um, you should have both levels of metrics, the differences in the level of details and the context in which each of those are used.

00:12:27

So if you look at flow metrics, their focus is that summary view for that value stream. And they aligned to the business outcomes. And more importantly, they are focused on predictability and efficiency and identifying where, um, strategic investments need to be made the software delivery metrics that we currently measure are great and needed to measure your day-to-day activities and focus on things like capacity, utilization, process improvement, um, and, and individual team improvements and so forth, and provide that deep dive layer, um, that on output that, um, can impact outcomes. Um, so if we look at what constitutes flow metrics, um, we have five core metrics that you can look at velocity, which gives you a sense for how much, um, how much fun are you adding work. You have delivered to a customer over a period of time. Efficiency talks about how are my processes doing.

00:13:37

Um, our teams are spending too much time waiting, um, before they can, um, complete their work. And where are those weights states, um, float time provides that customer perspective of what is that time between request to release and how quickly are we delivering value to customers? Um, load focuses on work in progress and can be a leading indicator of challenges within a value chain. For example, if there's a lot of work and demand for a particular, um, product and not enough capacity to take on that can a load can provide that insights to figure out what adjustments need to be made, um, to address that concern and distribution halves is one of my favorite ones, because it gives you a quick sense for how you are breaking down, um, your finite resources across the types of work that you do within a value stream across features, reef, defects, risk, and debt, much like the whiter signs that a doctor takes.

00:14:49

When you go, um, go visit them. The flow metrics provide that pose of the value stream. At that first instance, you can't rely just on those metrics in isolation, because in isolation, they're just pretty pictures. What you need to be able to do is align them to business results, anchor them in business results, without which, um, they're not going to be valuable at all. And this is absolutely important because the business cares about, um, to be able to align what the business is looking at as how it, how is the value stream performing against those results that we want to measure its performance on.

00:15:42

And another key factor is to keep it real time, um, stealer the metrics, um, not the, the, the meaning and the insights that you get is not going to be valuable enough to you. Once you get to a point where you have your metrics measured, some of the questions that come up, especially around load and distribution is the factor. Why does incise of work factor into load and distribution? Um, one of the key reasons is that size is a very, um, team specific, um, metric, um, team specific context. Every team does sizing differently, um, and they batch it differently. Um, the other reason is that from a customer perspective, sizes invisible, um, they only see what value they get out of a particular feature or a fix that has gone in. So why the size is important when you do your team level metrics? Um, it does not, it is better to focus on value when it comes to the flow metrics and the value, um, value is measured.

00:16:59

Um, when you look at the number of items delivered rather than, um, the size of it, let's take an example of Tasktop itself over a year ago. Um, we have to spend a whole cycle of a pie cycle on, um, architectural changes, which meant we had to, um, EMR a significant capacity on debt work, which left us with very, very few cycles for, um, feature work. So we decided to do what we call this quick win days over a period of two days. Um, we, um, identified all the features that have been in the back burner, which would have been, uh, which would have taken an effort of two to eight hours to be able to tackle them within a day. So we can pull that, pull that, um, pull those together. So a from our perspective, from the developer's perspective, this was great because they had, they had the opportunity to focus on deck work, which they hadn't been able to for a long time, but from a customer perspective, uh, based off 50 new features come in across buying components, pretty much every customer had at least two or three, um, features that they could, um, utilize, which meant our, um, customer MPS was through the roof.

00:18:18

Um, just so just, just to give, give you that, um, that sense for how it did not matter how long it took to complete each of those features, but each of those was quite important on their list and it enabled them to do their work on the product much better. Um, so this has now become a regular feature and most of our cycles now to be able to tackle some of these quick wins as we go. So the key there is that value doesn't always correlate to size. What's better, is to be able to prioritize what, um, impact business outcomes. And, and these are, these tend to be value units, whether it's features, defect, risk, that which will thrill and provide that benefit to the customer rather than the size. And if you want to gain this metric, reduce the backslides is because, um, if at a lower back size, you're able to deliver features to a customer and close that feedback loop quickly, that actually works for the benefit of the customer and for the product, and which is much more desirable, um, as much more desirable for that value stream.

00:19:43

The other concentration is that at the end of the day, if you have a lot of load, no matter what the size, you can only take on one at a time, one person can only work on one at a time. They would have to stop one to pick the other. So at the end of the day at a across the value stream, you have to, uh, look at the quantity or look at the number of items you have that you need to pull through from your backlog and the size generally at a value stream and flow metrics level is the less relevant sure. On the team level, when it comes to capacity planning, it is, but for the flow metrics, you're better off focusing on how many words, how big once you have flow metrics. Uh, the other question tends to be no, which one of these metrics do I focus on?

00:20:44

Um, it's, it's challenging because one, there isn't one metric that rules them all. And let's take an example of this customer who had pulled through their, um, flow metrics. And initially the, the first observation was they have great flow efficiency. It was in the nineties and top nineties actually, but suggested that they're doing great in terms of being, um, being less wasteful of people's time. There's less waste time. Oh, wait times. And they're, they're very efficient in getting things done. However, that story didn't quite correlate with their flow load because they tend their load kept rising. Their float time kept rising. Um, so it sounded like everyone had at his three or four items that they're actively working on does not make sense. Look, person can work on five different things or four different, excuse me, four different things at the same time. So as the look Dell, uh, delve into what those states are, and they, uh, utilize their, um, um, drill down metrics, uh, in the individual tools, they realized that their workflows were not adequately, um, tagging weight states, which meant that the flow efficiency was artificially up.

00:22:16

Well, if this customer had only focused on flow efficiency, they wouldn't have been able to see the problem. And what you need to be able to do is look at one metric and, but correlate that with the others. So they paint the same picture. The other key thing is to focus on trends, what works for one particular product won't work for the other. Um, you might have to focus more on defense for one, uh, um, older mature product was, is, um, you're probably pushing more features, um, um, a new product. So instead of focusing on absolutes, focus on trends, that gives them a better gauge of what's right for that particular product. Um, and finally, when it comes to frequency, uh, most before improvements, um, it's better to align them to business results, to identify how they demonstrate outcome, rather than focusing on setting, um, just levels on, um, individual metrics.

00:23:26

It's, it's what you want to be able to do is set a level on, say velocity, but correlated with results, which is much easier. Um, I'm much more realistic, um, to track if I had to put up one slide to summarize some of these points, I think it could be this, this would be that one slide you would want to take away. If so, um, yeah, so I think the crux here is that, um, D determine what the outcomes desired are, because that would enable you to identify the right level of metrics for, um, for your business partners, um, and anchor them to business results, whatever metrics you choose. If they have that business context, they, they paint the right picture, um, leverage flow metrics that are at a aggregated and summary level, but are correlated to, um, business results to provide that business visibility rather than throwing every metric that you have on a dashboard, which might, um, dive view the message that you're sending value does not always relate to size.

00:24:44

So focus on value units when you're looking at flow metrics and focus on size when, when, what you need to be doing as capacity planning and utilization planning and identifying, um, individual bottlenecks within a particular function or a team, and finally focus on trends because they are a better indicator. And don't focus on one metric, rather look at the, look at a few of the metrics together to make sure that they say they paint the same picture. Um, that's, that's all from me. Um, thank you so much for listening and have a lovely rest of the conference.