00:00:00

<silence>

00:00:13

Hello everyone. My name is Mick Kirsten, I'm the founder and CEO of Tasktop and the author of Project to Product. And I'm just thrilled to be part of this amazing learning community and to be sharing with you some of the main things I've learned since the sort of April May timeframe when I last gave a talk on tracking DevOps metrics around flow and really measuring objectives and key results or OKRs. So I hope to, uh, share you with you my latest learnings on how we go from micromanagement misery to finding flow for our organizations. And I have to say that this has been such a hot topic with many of the executive discussions I've been doing as OKRs has become a more and more important topic. At the same rate over the past year, I've been seeing a lot of failure modes around OKRs and a lot of organizations struggling to adopt them effectively and struggling to shift away from these ways that we've been accustomed to in terms of tracking activities instead of outcomes.

00:01:05

So the goal here is, of course, to show you how you can actually use OKRs to help you move from project to product, from, uh, outputs to tracking outcomes, and to help your organization move faster in terms of becoming a, a digital and product oriented innovator. Now, OKRs, for a lot of people, uh, probably started with reading books like John Do's Measure What Matters, uh, which in 2018 defined what's objectives and key results are, is these objectives, which are meant to be both qualitative, inspirational. They give us a way to point the direction around driving innovation, driving value to our customers, to the market. And then the key thing is having only two or five krs or key results, uh, which are in the end metrics of progress, how we track how we're doing. So it's, uh, in an interesting coincidence this week is the, my 10th year of setting OKRs for task stops.

00:01:51

So we're finalizing this particular week, and it's been a learning journey of over a decade to understand how we get better and better at it. We've made some significant changes, uh, in this particular cycle, and also learning to how they get adopted at massive scales and organizations that have tens of thousands of people relying on alignment around OKRs. So the key thing about OKRs is that the, again, what we wanna track is outcomes, non activities. And of course, we know that projects tend to specify activities rather than specifying, uh, these important outcomes for what we're delivering to the business, to the customer, to the market, to our partners. And this is where really the, the flow framework came in. The flow framework originated out of my own work within, within our own organization of understanding how to connect these business metrics to actual business results or key results and outcomes.

00:02:37

And what the flow framing basically says is that we need to understand, first of all, what we're measuring those outcomes for. And everyone working on that particular value stream needs to understand what outcomes they're driving, who the customer of those outcomes is. Now, the business key results, those tend to be well understood already. These are around improving things like net promoter scores or retentions or, or financial metrics. But the key thing I realized that was missing was a way of actually tracking how we get there. How do we know whether we're improving? How do we know whether the, the DevOps initiatives that we have underway are actually making it easier for our teams to deliver those kinds of business outcomes to the customer? And so the whole idea of these flow metrics is to actually have a set of flow key results that actually tell us whether we're improving, uh, or whether that we've got impediments on bottlenecks that are slowing us from improving, uh, in terms of how easily we're able to deliver those kinds of outcomes and to understand where we need to invest.

00:03:28

Are we lacking some DevOps automations? Are we lacking some important platform components? So idea, of course, is that we both track these business key results as well as slow key results. So really what I've seen when used effectively is that OKRs can help catalyze for you to shift from project to product. The challenge is, and I'll I'll speak to some of the pitfalls shortly, is when we snap into old behaviors while renaming them with OKRs. So here are a few of the, the bigger pitfall examples of, of Encounter that how things can actually go quite wrong with OKRs. So one of the main ones that I've seen is actually when we basically bring back the same behaviors, so the whole idea around OKRs to make sure that we we're empowering teams to set their own targets, to understand the outcomes and to have those aligned to the business outcomes.

00:04:14

However, OKRs will often get used as, as a way for micromanaging teams, for example, to just accelerate this feature and basically, you know, get to this point where OKRs are dates for things that that should actually be tracked on release plans and roadmaps. So we're moving away, away from, uh, how we should be doing Agile and planning, and how we should be tracking those activities and snapping back into effectively waterfall ways of planning. And this is one of the challenges that we see is when they basically OKR, uh, cascades become a whole bunch of projects and activities that tell teams what to do, we're completely missing the bone, and that's exactly what we should be using OKRs aligned to Agile as a way to steer away from. Uh, another really interesting pitfall that I've noticed across the board is when the only key results that are being tracked are the business and the financial metrics.

00:05:00

So metrics, you know, basically on cost. And of course, if we're only ever tracking cost, we again, we, we fall back into that cost center trap, uh, rather than tracking actually innovation and tracking how investments are driving business outcomes and outcomes for customers. So we somehow need to compliment, as you'll see in this talk, those financial metrics. Another even more interesting one that I've been seeing as a, as a common pitfall is to take team level telemetry and metrics, uh, such as, uh, uptime or such as, uh, deploys per day or such as user story point velocity or those sorts of things. And to say that those should be organizational key results. And of course, when you do that, when you take a team's telemetry, uh, which is very important telemetry for that team, and we should be measuring each of those things I just mentioned, in addition to, to all the other team level metrics, the agile metrics, the service, uh, and uptime and stability metrics and the DevOps metrics.

00:05:49

Uh, but if you're setting that as a goal for the entire organization around one of those metrics, you fall into this local optimization of the value stream trap, which we've all seen, which is that we're basically looking at, uh, uh, measuring a value stream with a two inch loop. This is, uh, John Willis said this beautifully, but measuring a a 12 inch value stream with a, with a two inch ruler, we, we need both sets of metrics. We need to understand what the in to play between them is. Uh, and then another very interesting one that I've been seeing even more is when OKRs are introduced in feedback cycles that are much too slow. So by design, of course, OKRs are, they came from Silicon Valley that around product oriented innovation rather than project management as the way that we operate, uh, digital and innovation. Uh, and when we've got cycles that are now taking 120 days, let's say, to deliver value and provide feedback, you can't even adopt OKRs.

00:06:39

This is something that with, uh, the OKR guru, Felipe Castro, uh, we, we realized in a webinar and, and the project to product podcast that we did recently is that when we're measuring organizations who can't basically deliver features to customers within 90 days, and when you're deploying OKRs there, your feedback cycle is just far too slow. So you can be in a situation where flow, the flow within the organization today doesn't actually support the deployment deployment of OKRs, which is of course, something we wanna move away from because this can be such an effective vehicle. So the main thing that I wanna do, wanna get across to you on this talk today is that in each of these cases, we don't have a, we don't have the, the three ways of DevOps in place, the flow feedback and continual learning needed to adopt OKRs, which in the end are a learning vehicle for the organization.

00:07:23

They speed our decision making, they help us pivot and make adjustments faster. They help us experiment and test hypothesis faster. So in, how, how do we move away from that? And I think the fundamental think here is to actually understand flow and to understand that when we have key constraints that are not measured where we can't improve them, we fall into these traps that make it too difficult to adopt this fast pace. Basically this, this agile, uh, at planning and goal setting methodology. And so right now, I'll actually just relate to you a lot of the experiences that with, with some data. I, I shared some of the stories of what we've seen over the past year of organizations who've actually adopted the, uh, uh, a shift of value stream thinking. So are starting to shift from project to product, but really have starting points that, that are often challenging.

00:08:10

And the majority of these organizations, the majority of this data set that we collected through test desktop vis, uh, is actually from organizations who are adopting OKRs. In some cases very similar methodologies. But here's some of the examples. So in this case, the, the common fact that we're seeing across this substantial data set is that only 80% of what's planned by agile teams gets delivered. And as you can imagine, you can have the best set OKRs, but if that, that's the ratio of what's being planned, what's being delivered, there's just a very big mismatch between the capacity of the organization to deliver on the OKRs, uh, and the OKRs that are being set. So somehow these things, these things have become very, very disconnected in a lot of organizations. Here's another, this, this was a very startling one to me as I saw how, how significant this was across the customer base, is that only 20% of feature, sorry, 20% of features are canceled after code has been written.

00:09:00

So what's happening is you've got these plans coming in, and these goals coming in from the business driven by customers, given by demand and the like, and scopes are changing that frequency, that 20% of actually capacity that's been started is being thrown away, making of course, flow load and work in progress with challenges even bigger. So again, a really big disconnect between how planning is being done and how we're delivering, how we're actually, uh, tracking those activities and, and able to d deliver on what those plans are. Uh, 35% of product value streams that we're seeing have no capacity for new work. So again, really big disconnect between, uh, the capacity of value streams and the way those value streams are being planned, because of course, there's an assumption that there is capacity for new work, otherwise, why are we planning tho those kinds of outcomes?

00:09:46

Uh, 85% of products under invest in security and debt. So again, this is another factor that just exacerbates the deployment of things like OKRs when there hasn't been the, the investment in the improvement of work to actually take on the new work that's, that's being planned. Uh, and finally, 95% of value streams don't know what their actual efficiency is, and in many cases don't know what their capacities and the capacity, capacity is not being communicated to the business, to the planning cycle. So this is just some of the ground truth that we're seeing out there. And of course, the, the whole goal is how do we improve on these numbers and then get to the point where we've got a very good alignment between what the work that's being planned, the work that's being delivered. And of course, most importantly, that feedback loop. And really that's the goal of, of, of approaches like OKR, is to establish an effective and fast feedback loop between business planning strategy, uh, understanding and measuring and delivering to the customer base and what we're delivering through, through our daily work.

00:10:42

So the thing that strikes me the most is that measuring flow is, is how often it has not been part of how we plan and how we work. And I think the, the really neat things I've actually gone back to and, and most of my presentations to leadership teams gone back to, to this, to this, uh, tried and true, true quote from, from Gene Kim and the Phoenix Project that improving daily work is even more important than doing daily work. And really, if I could just pull out, you know, state one challenge around the way OKRs are being deployed today is that they ignore the improvement of daily work. We're asking our teams to do so much to change so much, their backlogs tend to be so large, and if we don't add capacity to the improvement of daily work, we will just, we will just overload them even more and get even less out as you saw in some of those last statistics.

00:11:30

So really, the, the question just becomes how can we use OKRs to drive the improvement of daily work? And of course, to really do that because OKRs are meant to speak in terms of outcomes, those key results that we, that we are delivering to the customer, to the business, uh, how we actually connect improvements in daily work to those business outcomes. And so I'll share with you just a, a few short examples of, of how this, uh, how I've seen this work successfully in organizations. But first, we, we do have to get on the same page of how to measure improvement, how to track flow, uh, across the technology, the business stakeholders, uh, and, and the way that we measure and that we, that we operate. So fundamentally to improve flow, we need a way of measuring flow that we can all agree on. And there's a really quick, quick recap for those who haven't seen it of the flow framework.

00:12:12

You can see more on flow framework.org. The question of course is how do we measure what, what flows in software delivery so that we know if we're improving it, we know if we're making the daily work of our teams better, we know if we're removing burden from them versus adding burden to them. And again, of course, the whole goal is as we're doing that, as we're creating these new approaches like deploying OKRs, that that's supporting this improvement of daily work, not just putting more on the teams. So very quickly, uh, what flows in software delivery according to the flow framework is features, defects, risks and debts. And so all work, all issue types and your agile tool map into one of these. And these things are a zero sums game, the mutually exclusive and collectively exhaustive. That means if a value stream is, uh, able to take on more feature work, chances are it's able to drive more outcomes.

00:12:53

However, there's too much technical debt. Uh, what's happening is that that, that that will take away from the capacity of the value stream, or if defect rates go too high. Incidents, there are too many incidents because of a lack of investment in, uh, in stable platforms. Of course, that will take the capacity of the value stream. So the point of the flow frame is to, to basically to expose and to make visible, uh, the zero sum game that we have on every value stream. And then to allow us to create organizational objectives around improving that in addition to creating objectives around revenue and cost and, uh, and customer outcomes. So here's a very quick example of how we can connect those things. Uh, first we need to make sure that we can measure these metrics. So just like you've got these four flow items, they're four flow metrics and flow distribution.

00:13:34

So that's flow velocity, basically how much we're able to deliver across the period of time. But the key thing is from the business, from the customer's point of view. So this is end-to-end velocity, not how long it takes one team to do it, but all the way from where, when work enter the value stream to when it's done. Flow efficiency, where a bottlenecks flow time, how long did it take to deliver from end-to-end all the way when work was committed, uh, uh, to be delivered to when we had running software, the flow load, that's just that, that wi metric. How much work in progress is there? So as an example, and now we've got a visual of a value stream over here. What we see is a value stream that does have a bottleneck. It looks, it looks fairly clogged, and, uh, some very bad o badly structured OKRs clogged it.

00:14:10

So when OKRs are basically around, get me this next feature, where is my feature? What happens is too many features get put into the part, this particular value stream, this, this set of teams that's, that's delivering value, uh, for a customer. And when they're being micromanaged, that causes a whole lot of thrashing, that canceled work that you saw, that 20% of work being canceled, that's exactly what, what's happening when that work is canceled and additional work is put in. Uh, and we've got everyone basically, uh, struggling to keep up the work while new work is coming in. And really the cause of this is that, that that planning, that demand is not connected to the actual flow, the actual capacity of the value stream, nor is the understanding of what the bottlenecks are part of, of actually the planning. And that's really the challenge, is you keep cramming more into the pipe without improving the, the, in this case, uh, the width of the pipe, and you get these predictable and problematic results.

00:14:59

When you're ignoring capacity, you just keep increasing work in progress. And this is one of the biggest findings that we've seen is that basically overloaded value streams are just endemic across enterprise organizations. Now, good OKRs are actually an opportunity to improve that. So bad OKRs make it worse. Good OKRs are, are a great opportunity to improve that. Uh, one of the main things they do, of course, is to cascade business outcomes to the value stream so that everyone can be, can actually understand, well, what outcome are we trying to drive to? Are we trying to, to capture more customers? Are we trying to keep our customers happier? Um, are we trying to, you know, grow our customer base? Are we trying to make a business partner happy? Are we trying to make a an API more stable? Because so much of our business builds on it.

00:15:36

Uh, to do that they need to measure the, we need to measure the flow of value. We need to understand where bottlenecks are happening and how we can actually improve on the situation to have capacity from more work. Because fundamentally, most organizations are actually constrained by the capacity of their value streams. Uh, and then of course to prioritize learning and improvement. So we're always making sure that, uh, once we found one bottleneck, we're then learning that where that next bottleneck is and then improving there. So I'll give you some very quick examples of, of how meaningful, uh, this approach can be. This is an organization, uh, it's a financial services organization, so it's quite a, a sizable bank. And in their case, as part of what they strove for in their key results, they wanted to just improve time to market. They want to reduce flow time.

00:16:19

So they were able to make that an, an objective, um, measure it through how many days for features flow time improved by, they saw a reduction from 55 days down to 38 across the organization. So this is a very, very side by, of course, by removing some bottlenecks, by implementing, uh, some DevOps automations, some security scanning automations and, and fixing some upstream bottlenecks that they had. And this translated this, this flow of this flow key result that was achieved translate to an $800 million revenue pull forward because of all the capacity they had to deliver on those plans of digital experiences that they had. You know, they had so many of their business cases and their digital transformation were based on, uh, another ones reducing the cost of quality. In this case, of course, you know, many of our of us are working with sort of very large legacy systems.

00:17:05

Uh, some of those have caught, you know, frequent outages and incidents in this case actually investing in, and this is sort of nothing new, say, uh, back to the future kind of scenario that we see all the time is actually investing in testing automation, test harnesses and platform stability, uh, unlocked basically by reducing defect, relu caused defect resolution time to be reduced by 70%, made their lives easy, much easier for those teams. And that unlocked $52 million of, of revenue growth through just a much healthier customer base. And then this is the, that story that we so often see, uh, large securities organization, and in their case, they were able to drive a 40% acceleration of feature delivery by reducing tech debt. And this is a, another really important scenario where they had this, this measurement of, uh, investment in technical debt and the outcome of that being additional feature delivery, and then the outcome of that additional feature delivery capacity from effectively, uh, the same size of organization driving this $140 million revenue pull ahead.

00:18:05

So again, you kind of get, get a sense of how we can use these objectives and key results, uh, and measuring flow as the key results, uh, as leading indicators of these business outcomes that, that we're all after for the organization. So I think the key thing now is that we understand the different types of metrics and the different things that we can measure the key results to get to those kinds of results that, that we just saw. Uh, so first of all, the business metrics, they tend to be already very well understood in, in every organization. I think in, in, and, and this is really for the business facing parts of your portfolio. Uh, I think one of the key things around the shift from project to product, of course, is to provide those business metrics for the platform products, for the shared services, for the delivery of CICD pipeline itself.

00:18:47

Each of those need some kind of outcome metric. How many developers are on the new pipeline? How many, uh, how many of the business applications are using the new APIs or the new cloud environment that created, we created in so long? But those do tend to be well understood today. Now, the key thing with the float, but the flow metrics is they provide a leading out, uh, leading indicator to how we're doing on that. So for example, if we're able to deliver more features with less toil, less burden, more quickly, we should be driving that, that customer outcome, we should be driving that business metric. And really it's that interplay between the flow metrics being a leading indicator and the business metrics, they tend to be a lagging indicator because to drive additional revenue to drive costs down, uh, tends to, you know, happen in many month cycles.

00:19:28

Whereas the flow metrics you can actually bring into your monthly review cycle, your quarterly review cycle, and understand whether things are moving fast or not, is also an example. And then the other key thing is, is the team metrics do not throw away the team metrics. This has been just a fascinating thing where organizations think, okay, well, deploys per day are no longer a representative metric for us. We've, you know, we're now at the point where we can deploy multiple times a day. You still need to measure how many times you deploy per day. You still need you, you need very much need that Dora metric. But chances are that you need a bigger picture metric to understand where your bottleneck is upstream of downstream of deployments. So again, these metrics are all meant to work together, and the flow metrics are, the whole goal of them is to be end-to-end, and to be cross-team cross value stream, rather than focused on, on improving one part of the value stream.

00:20:10

Now, of course, when you find that bottleneck, it's all about those particular metrics. If that bottleneck is, is in the kind of this, the safety and reliability of your deployment pipeline, that's what you focus in on. And those are the metrics you can improve as part of your, your quarterly KR for that, that part of your portfolio. Uh, another key thing of course, that I did want to relate is, is, uh, is, is roadmaps and release plans, right? We can't, just the same way, we can't, we, we should never throw out our, our team metrics. And those remain as important roadmaps and release plans and OKRs need to work together. So roadmaps and release plans basically define what gets delivered in, in which order. So I briefly flashed that, um, that visualization of the Sues canal in terms of us needing to think about improving flow and what happens when we have big bottlenecks.

00:20:52

So we can think of that as sort of the o order of the container ships that that now needs to go through that canal, needs to the, uh, needs to, uh, be delivered to customers in order to minimize delays and and to maximize outcomes. So those are the roadmaps, but roadmaps alone are not enough. They need to be aligned to our OKRs, but in the end, we need value stream OKRs around accelerating flow, accelerating how quickly we can deliver on those roadmaps and finding those bottlenecks, finding like, like the evergreen stuck in the, in the sue canal. So really think of value stream OKRs for every value stream. And it really is important to think about them for each one independently is how do we widen that canal to get more ships through? How do we make it easier to get ships through? How do we make it safer, uh, to, to get ships, to get ships through that canal?

00:21:32

So really, again, in addition to looking at the roadmap, which defines what will get done, which then of course delivery of that will define the the customer and business outcomes that we have, always looking at how do we make it easier to deliver? How do we make it better deliver, how do we get to those, uh, to the five ideals from the unicorn project, um, in our daily work? And again, just becoming measured around it. So where OKRs are a great tool. And then the other key thing is the organizational OKRs. And this, these are really the, where transformations and large scale digital transformations can use OKRs to help track the improvement of work. Because one of the big challenges that we have, of course, is a lot of large transformations have taken a completely waterfall approach to their agile transformation, ironically, or the, or the DevOps transformation or the digital transformation as a whole.

00:22:15

And in the end, we wanna be able to measure that transformation. And under, and this is, so the visual i I I have here is to how do we build new canals in terms of the transformations? How do we build that, that new canal, that new cloud platform that we move a lot of our business applications onto a lot of our customer facing applications onto? And then how do we track whether we're getting the right, right results from that, whether that canal is working, whether we dug it too shallow, uh, whether we have some, some other unforeseen circumstances that means it's not as productive as we thought it was. And so understand those organizational OKRs, and I'll show you an example of all of this in just a moment, is, is key to, uh, measuring in an iterative way and applying flow and feedback and continual learning, the principle of dev principles of DevOps to the transformation itself.

00:22:58

So here's an example from an insurance company, uh, and one of their OKRs was to become the most innovative insure in their industry for this large enterprise. So they had a relatively small market share, there were some larger insurers there, InsureTech companies, um, and they wanted to grow their market share, um, from six to 30%. So the really sub substantial part of the market over the course of, of the, their planning window, which in this case was a year. So it was really significant stretch. Uh, and then they created this beautiful and aspirational, uh, KR to reduce the time that it's took to provision policy to basically cut it in half from 43 to 20 days. And so this is all about the customer. How can we get the customer, their home insurance, their car insurance, whichever part of the business this was, how can we get to them in half the time?

00:23:41

That kind of cascade down to everyone, every value stream supporting this, every team supporting this is extremely valuable because it makes you rethink the, uh, technology architectures that you're using. The sign on the, the, the authentication, um, the sign up, uh, approach that you're using in case that's the bottleneck to the customer's journey. And then finally, this, this last key result that they track was to improve flow time by 20%. And this is actually how they capture the need to drive improvement within the value stream itself. So it's that last one that's so powerful because again, it's not just prioritizing work, it's actually helping us improve daily work. And so here's how some of the value streams, I'll give you two examples, actually pick this up. So this is the, the mobile application value stream. Uh, what they said is, for us to become most innovative customers have to love our mobile experience.

00:24:26

Our star ratings are not great right now. We need to improve our MO mobile experience. So they translate that into increasing their net promoter score for their mobile applications from 31 to 60. So they, that's a pretty high net promoter score. Um, and they, they were really want to make sure they could do that. But to do that, of course, they realized they needed to get more features, those features that delighted the customers that made it easier for them to authenticate and to, to buy insurance products. Uh, they needed to get that feature full time down from what was over 20, what was over 20 days to under 10 days. So basically took cut in half in 20 days is not bad. This organization's doing quite well. Of course, we measure flow times in, uh, in months quite frequently, but still it wa they realized it wasn't quite enough for them to drive the sort of result results that they wanted in the timeframe they wanted.

00:25:08

Now, from that flow efficiency, uh, from improvements, they had the organization level, uh, they already had multiple experiments from those exp uh, going to understand where the bottlenecks were from those experiments. And the way that they were measuring those, they realized there's actually verification of new features by their business partners that were the bottleneck. The bottleneck was not the teams, the deli development teams themselves. So they set this aspirational, uh, goal here of saying, we're now going to target and this all happening within one quarter. We're now going to target zero days wait states on business input. And what happened as a result of that is that the flow time went from nearly 30 days to well under 10 days on average. Four new features that they were delivering, uh, net promoter score started climbing. But the net promoter score was a lagging indicator of this because what happened here was that the net promoter score, uh, was a result of delivering those great features.

00:26:00

Now, of course, Chan, if they could have delivered those great features and maybe the net promoter score didn't improve, then that indicates there's another problem. There's a disconnect between the investment flow and then building those great features and what's driving customer outcomes. If customers have already left and gone to a competitor in this case, they were able to actually see that correlation, that very direct correlation between improving flow and driving those business outcomes. And so this is exactly the kind of feedback loop we wanna put in place to make sure that the activities that we're doing and how we're improving that work is actually then driving the, those customers, those business outcomes and then feeding back into planning cycles. Because of course, as you can imagine, this was a really major success for the organization and taught them about the importance of, of faster feedback loops.

00:26:42

Uh, here's another really quick example, and this is a, this is now a platform team, so it's not a customer facing team. And in this, it's actually the same organization in this organization. They realized that they had done such a fast lift and lift and shift to move to, uh, basically their, uh, some of their core platforms and systems to the cloud, uh, that they were making very inefficient use of storage services in the cloud because they, they were not using modern storage services that they'd moved a whole bunch of their databases effectively to, to be hosted. And so the per application hosting costs were certain to look prohibit prohibitively high because, uh, and part of the ROI of cloud was meant to be that we actually improved our cost profile, didn't make it dramatically worse. Now, what the teams knew at that point was already was that they had, again, in this very fast move to cloud and what was too much of a lift and shift, they brought on a whole bunch of technical debt because they were not leveraging those new storage services, uh, that were actually available from their hosting provider.

00:27:39

And so they basically spent a release cycle where their flow velocity was dedicated to tech debt reduction and they tracked both that flow metric. And on the left here we see how they're tracking their application hosting, hosting cost as a kr that's a key result. And they saw as they took down that tech debt moved, those, the new storage services, this hosting bubble was dramatically reduced and they were in a much better point for scaling and bringing more, uh, of the business onto this new cloud platform. So again, whether we're sort of focused on top line or customer results, or in this case actually just, just the cost structure, uh, that that will, you know, that will happen when, when we're not keeping up with technical debt or not leverage new technology platforms as as we had intended. Uh, this again gives you that, that feedback loop of connecting that that investment flow in tech debt reduction to actual business outcomes.

00:28:25

And so in summary, uh, the, I think the, the, the, the key point here is to make sure that you are as if you're leveraging OKRs or similar planning systems. 'cause of course a lot of us have similar planning systems that aren't quite the same. Uh, you can lev use those to catalyze your shift from project to product. And really the, the, the main part of that is to make sure that you, as you focus on daily work, as you focus on improving your roadmaps for, for every product value stream as you make your internal products have their own, uh, first class roadmaps, uh, your platforms have the roadmaps that you prioritize and measure both daily work and improvement in daily work. And so that every value stream always has that flow metric, uh, as a key result. So those flow metrics can really help you prioritize and keep and, and track that improvement.

00:29:11

And as you saw in some of the examples here, actually see a very fast payback loop on that improvement. And so they empower value streams to set their own flow key results, uh, as, and then of course connect those to those business outcomes. And I think the key thing I've also learned is to, to celebrate those successes when you've established that feedback loop, you're driving those outcomes that you let the rest of the organization know in terms of help. I'm looking for if, if you're seeing that, I'd love you for you to share that out as well. Um, on the DevOps enterprise channels are, are on flow framework.dot org. So, uh, to learn more, uh, check out project to product do.org or check out the project to product book that speaks a lot about this. And remember that all our proceeds go to supporting charitable programs, uh, for women and minorities in technology. So with that, thank you.