00:00:00

I hope. Uh, and I'm so delighted that so many of you have indicated your interest in our first re-imagined DevOps enterprise community event. So within the program committee, we are deep in the process of creating the DevOps enterprise summit, us programming. And I am so excited about how it is shaping up. We haven't announced, uh, many of the speakers yet, but I can tell you with complete conviction that, uh, it could be our best programming we've ever, uh, put together. So, uh, why are we here today? Um, it is, uh, literally true that since 2015, uh, there have been discussions and wonderings from, uh, all of you and within the program committee about how we can continue, uh, the amazing interactions, uh, between, uh, between the conferences. Um, uh, in 2015 we had our first discussions of this and, uh, we've tackled this, or we try to tackle this for years and, uh, in our kind of twenty, twenty one, uh, planning kick-off call within the program committee, uh, Ben Grinnell from north Highlands said, uh, what an amazing opportunity to quote, stop force feeding attendees twice a year, and then starving them the rest of the time.

00:01:07

So, um, rest assured I know that there's a, a need a want for this. Um, and after all, one of the top things that people tell me, uh, after, uh, the conferences, whether it's virtual or physical is what an amazing three days I feel re-energized, and hopefully energized enough that I can keep up the good fight for one more year. So we've made a couple attempts to do this in the past, uh, in the past years, but that we've never really quite stumbled onto the right formula. Uh, but at the last, uh, development of our summit, Europe, Nick Egleston, and Jim, uh, and James mover Lee volunteer to help us explore this further. And so I'm going to be having Jim, uh, share a couple of words on why this is important for him, some of the things that he's trying to achieve and maybe some help, uh, that he's looking for.

00:01:51

And he will actually be hosting an after party, uh, after this, uh, planned 90 minute event. So, uh, before we go to, uh, Jim or James, um, uh, here's what will happen the next 90 minutes, we will be having, uh, three panelists today. We will be having, uh, John, uh, smart, um, partner at sooner, safer, happier cursed, Dr. Mccarsten, uh, CEO of Tasktop and, uh, Julia Harrison, uh, from GDS, um, uh, head of product at govern digital services. Uh, talk about OTRs. Um, and so how, in the next 90 minutes, we're going to have a combination of a watch party. Uh, we'll be watching a video together. Uh, we breaking in for questions, uh, either a planned question between, um, that we came up through the panelists or questions from you. Um, you'll have the opportunity to join, uh, as a panelist. And so the goals are one learned ton, uh, to, uh, hopefully, uh, move the needle in terms of, um, our goal, which is to help technologies leaders succeed and the organizations win. And then specifically for this event, create enough exothermic energy to have everyone say, ah, let's do it again next month. So, uh, with that, uh, James, I'd love to turn it over to you and, uh, have you share your thoughts in your own goals?

00:03:10

Thanks, Jean. That's a real pleasure to be here guys, and I'm, I'm literally blown away by the engagement. It's been a fantastic process. Um, I attended the, uh, the DevOps summit in may, um, and I'm just early in my, my journey, uh, Jean's journey, uh, read the Phoenix project that they've ops handbook see to safer, happier, and I've been to onboarding and, uh, I work in a very old sort of state of telco industry. I worked for BT BT itself, a very large company in many different facets. Uh, but I specifically work in the telco area and they're usually about 10 years behind the rest of the world. So some people might resonate in that. Um, but yeah, during the DevOps summit, I think brilliantly Nick Eggleston, um, my fellow, uh, attendee asking, Hey, what do we do between the get togethers? Right? And so Jean reached out to us, uh, asked us to get involved.

00:03:59

We've heard of hear discussions with them. Um, and when asked what I look to get out of this, it was like, yeah, you know, I'm midway through my career. I recognize this as an awesome, uh, effort by these guys to do something different, change the way we work, look at the technology, look at the way we are as humans and how we work and coming to the, uh, coming to the festival or the summit really gives me energy to, to fight that flight back at the office and deal with the management and deal with the people and deal with the mindsets. And yeah, so more of this is a good thing for me, for sure. And I, I really look forward to meeting other like-minded individuals, uh, which I will be at the bar afterwards. So please come and join me, come and say hi, come and join in the chat. Um, quick word about community, please. Don't be a taker, try and be a giver, try and participate, try and ask questions. These good people in front of you have written some great materials. Let's not drain them of their energy. Let's try and give back a little bit. So yeah, hopefully some good questions and comments in the chat as well. Anyway, I'll let you guys crack on. Thanks, Jean.

00:04:55

Yeah. And thank you, James and Nick, uh, for putting so much, uh, energy enthusiasm and some focus, uh, to make these things happen. All right. So let's go to the next stage. Let's talk about, uh, let's introduce our panelists. Um, so, you know, just to put some context, uh, we had, uh, as James mentioned, we had some, uh, calls to explore what should we be doing in our first, uh, community event? And, uh, the idea of a watch party came up and the notion of mixing the, kind of the cinematic experience of watching and learning through a talk, but also doing something that you can't do, uh, at least easily in a, uh, in a live event, which is actually have interactions. And so, uh, when the topic, um, the question was asked, all right, what topics should it be? Uh, w it seemed pretty obvious that the two talks, um, uh, that were some of the most, um, uh, widely received, uh, were about OTRs. So my, to my panelists, uh, to John, to Mick and to Julia, uh, why our OKR is so relevant now and why it's so important that we help them succeed. So in any order, I'll pass it over to the panelists.

00:06:04

Oh, I'll go first. Uh, so, um, why are OKR so important? Organizations are embracing more modern ways of working we're in the age of digital and OKR is help with the sh the pivot for a focus on output and the fixed solution to outcomes. So it really helps organization focused, focused on outcome thinking, and, and also kind of limiting, uh, you know, that you can't prioritize everything. If everything's a top priority, nothing's a priority. So it also helps organization to articulate what are the top three to five things we're going to work on.

00:06:49

Awesome, Julia.

00:06:52

Hi. So, um, I think I've got a little bit of an unusual take on OKR, which is most of what I've read and heard about are okay, as we're talking about from an organizational point of view, we're talking about how to drive from the top around setting objectives at the top of the organization, and then cascading those down through teams. And my own experience has been that actually, if you're in a team, who's starting to do agile things in an organization that is not fully agile from front to back, or maybe has a ceremonies, but isn't quite doing that outcome focus yet, um, by focusing on an objective and getting your nearest stakeholders to agree to an outcome based objective, rather than build me a thing. It's one of the first ways you can carve out a bit of autonomy for your team. So it's a really useful tool in driving kind of agile thinking from the ground up in an organization. That's not quite getting it from the top down yet.

00:07:45

And by the way, the feature is a, of course, a workflow status. I loved your, uh, I learned so much from your talk about notifications make,

00:07:54

Yeah, I guess I could not agree more with both points. I think that the interesting question for me is also why now, because I was as surprised, I think as a, as I know you were initially, and, and John, like why this became such a hot topic so quickly in a DevOps enterprise community. And it was particularly surprising to me because I've been doing them for this my ninth year of doing OKR. And I started, I thought they were popular at least in startups at that point and in the bay area. And then they've been used by Google and others, but then until before that. So I think the, the why now is, is really interesting because, uh, I think it's just a sign of how much organizations are maturing in terms of how they think of dev ops, right? Uh, if we think back to the Phoenix project, it was about this end-to-end flow into when feedback and learning, and you can't do that.

00:08:39

If you're working in a mode where all work is thrown over to it, you don't really have to think about outcomes. You're telling it and agile and technology teams what to do, and the business and the people with the MBAs, you know, sitting in a potentially different set of offices, uh, are worrying about the outcomes and, and th the technologists don't need to see them or know them. And I think this is a really fundamental shift because now of course, everyone's realized how dysfunctional that is. And unless we all get aligned around customer and business outcomes, we won't succeed. We won't create the kind of innovation, uh, that startups then that technology companies have have seen. So I think it's a, it's just a really good sign, this, the shift to outcome.

00:09:18

Uh, last question, before we move into the first video, uh, is the planning call a topic came up, which is, uh, we can't let OTRs fail that, uh, that, oh, Kara is in a hype cycle, uh, that may or may not be true, but, uh, okay. I was, are a good thing. Um, and, uh, we can't let poor implementations and poor thinking, uh, uh, screw. Okay. Ours up. Can any of you comment on that?

00:09:44

So, so the, um, there, there is a real risk that with the hype cycle organizations do OKR is for the sake of doing OKR and it, you, oh, we doing okay. And you kind of, there's a risk that organizations have lost the reason why we're doing okay, ours, um, and related to that is some organizations and some teams find it quite hard to prioritize. And the thing with OKR is one of the beautiful, really positive things about the mindset without chaos. It, it means you have to prioritize, you have to say, what are the top three to five things that are the most important things, be everything you're doing? So the, the, the, the anti-pattern the thing to watch out for here is that if teams are not very good at prioritizing, they blame OKR. And it's like, okay, ours don't work for us. You know, we're going to have to throw away okay, ours, but actually it's because the team or the people actually are not very, you know, need to work on their ability to prioritize.

00:10:54

Right. Any other comes on that, otherwise let's start our first video,

00:11:04

Just jumping quickly here, because I've, I've, I've seen this already, right. I've seen, I've seen my colleagues say, you know, please don't talk to me about, okay, ours in large organizations. We've like, we've been, you know, it's now been a year and it's just KPIs all over again. And it's, you know, it's just the latest trend here. So I think the really important thing is that that OKR is our, again, in terms of, it's just a better structure for planning, for some of the reasons that we just heard that gene alluded to, right? And, and as John said that you're living to three to five, they're made out of cascades and they really are made around measurable quantifiable outcomes. So my concern is absolutely that we take this opportunity of actually aligning everyone around customer outcomes around this fifth ideal. Uh, and, and that we must it up and it just falls to another, another management methodology.

00:11:47

And then they degrade back to KPIs. And I think one of the biggest dangers is, is that as these things are being set and rolled out, uh, we fall back to activities being the way that opioids are being implemented. And there's really no changes as John pointed out in, uh, driving prioritization. Because when you think about comes, you have to make trade offs, you have to do this prioritization. So we just can't let it snap back into the old ways of planning, uh, or, uh, does I've been concerned snap back and do waterfall processes back between the business end of the technology teams. So I think it's, it really is important that everyone listening, who cares about this topic helps their organizations not, not have, okay, ours fail as another, uh, another management fan. So

00:12:25

Brilliant, Julia.

00:12:28

Um, I think as well, it needs to feel useful to everybody in the, in the chain. It can't be a management is doing this thing to us because they need some reporting. The teams have to be involved in setting their OKR setting, okay. Which they think are meaningful in terms of the work they're doing and what value it delivers to whoever their customers and users are. Um, everybody has to feel it's useful to them in terms of inspecting their work at every level of the organization. Otherwise it becomes a set of reports. And as soon as managing the work becomes divorced from the reporting, you'll get reporting. I mentioned to me at myself there you'll get reporting. She's always green and the managing the work will happen somewhere else.

00:13:13

Brilliant. The TPF forms from office space. All right. Um, I am going to attempt to start the video, um, standby two desktop watch party and John's video. All right, here we go. Fast. So many of the principles and practices he's developed over the years, it was informed so much by his lessons learned leading the ways of working at Barclays, a bank founded in the year 1635, which actually predates the invention of paper cash. We've heard so many problems, statements being voiced by this community about problems being encountered during the OKR planning process. So John has some pretty brilliant observations on what makes for good OKR versus what makes for bad. Okay. Ours, and some incredibly practical advice. I'm a huge fan of John's work. And I think this might be some of his best work yet. So here's John.

00:14:21

Hi, my name's John smart. And I'm going to talk today about chaos lessons learnt in the implementation of OKR is around. Oh, okay. Okay. And not, okay. Okay. RS and the three M's, which are mindset, mission, and measurement. First of all, we will take a look at what our, okay. Ours. Well, then have a look at the anti-patterns and patterns. The lessons learned the hard way. So the, the, uh, the patterns are the Oko and the anti-patterns of the knots. Okay. Okay. These are, like I said, lessons learned the hard way I have experienced of implementing OKR across tens of thousands of people with multiple business units and thousands of products. Um, as well as supporting organizations across different industry sectors in their adoption of OKR, we will then take a look at, okay, our alignment and in particular, this is relevant when you get into size, scale and complexity.

00:15:22

So how okay. I was aligned and we will take a look at how to get started or how to get we started. But before we do that, let's take a look at the rising popularity of OKR is taking a look on Google trends. I looked at the, the number of searches for the term OKR and the number of searches for the term balanced scorecard. Interestingly, around about the middle to the end of 2018, there was that tipping point where, okay, I became a more searched term and balanced scorecard also, uh, quite what surprised me, uh, pleasantly surprised me is that there are now more searches for the term OKR and their offer portfolio management. And again, that tipping point was towards the end of 2018. So pretty recently we see this, a shift in consciousness around adopting. Okay. And I'm sure it's absolutely no coincidence that the, uh, the rise in the trends that coincides with the DevOps DevOps enterprise summit, London, and Las Vegas, um, a symptom of the hype cycle around OKR, if you type in OKR certification into a search engine of your choice, you'll get pages back of organizations who are very happily take your money to make you a cop, a certified OKR practitioner.

00:16:48

So, first of all, what are okay, ours? Okay. Our objectives and key results. And there are three M's for OKR ours. Uh, I like to think about them with the three M's they're all equally important. The first M is mindset, and this is having an emergent mindset over a deterministic mindset. This is acknowledging that the future is unknowable, and it's also adopting a stance of empowerment, empowerment, but also taking empowerment. Second M is mission. This is the objective, and this is outcome over output, and it should be both inspirational and aspirational the icon on the left. There is a flag and effectively, this is planting a flag at a point in the future, uh, for teams to then experiment as to how to best achieve that goal, that target that mission.

00:17:45

And then finally, the third M is measurement, and this should be measuring movement and behavior. So these are the key results. And if you're not measuring movement and behavior, you don't know whether you're actually going to achieve your outcome. You don't know if you're making progress in particular behavior. If behavior doesn't change, you're going to get the same outcomes so that there has to be a change in human behavior to have a change in outcome, and they should be leading and lagging measures. If there aren't any leading measures, again, you're like you it's like you're flying blind. You've got no idea. If you're making progress on your desired outcome, um, the better you're leading measures, the more agility you can have to be able to pivot when we look at the history of, okay, ours. So, okay. Ours came from embryos management by objective. Uh, embryos came from Peter Drucker who wrote about them in 1954.

00:18:42

And management by objective has been the, the almost universal way that organizations, large organizations have organized what work gets done. So the characteristics of management by objectives, they are top-down. They are generally focused on output and tasks. They are static, typically annual. They are private and siloed. Typically the objectives are tied into your performance appraisal, they're in the performance appraisal system. So of course, they're there, they're private. They're not available for people to see in that context. And because they're tied to your performance, appraisal, your pay and everything else, they tend to be risk averse. It tends to be a case of shooting for mediocrity, because you'd rather under promise to be able to over-deliver. And finally, the prevailing cultural norm is one of command and control and benevolent dictatorships, where the objectives are, are handed down from the managers to the workers that tends to be the norm.

00:19:48

And that tends to be the starting point. So in terms of OKR is, um, it's Andy Grove at Intel who took embryos and he, he, um, evolved them into what today. We, we know as, okay, ours, he called them. I embryos Intel embryos. John Doerr worked with Andy Grove at Intel, and he took the concept to Google in 1999. Um, you know, and, uh, Larry is quoted as saying that he attributes the success of Google to the fact that they used, okay, ours today, it's companies large and small unicorn and not unicorn who are using OKR. So what are okay? They are top down and bottom up. They are not just a top-down cascade, but they are top-down strategy meets bottom up teams and team of teams articulating their own outcomes, aligned to the higher level strategy. The focus is on outcomes and experimentation, as opposed to outputs and tasks, acknowledging that the future is unknowable, focusing on the outcome to achieve, and then running experiments to try to achieve that outcome.

00:20:58

They are more dynamic. They are multi-year annual quarterly. They have alignment, you'll get feedback. You might get daily feedback, and you can feed that back into your outcome hypothesis, what you think of the right outcomes. And, and you can pivot, you can exhibit genuine agility. So they're much more dynamic, they're transparent and aligned. So you should be able to see the OKR that other teams are working on. So you can see what people are working on. You can clearly identify overlaps or duplication, um, and you can collaborate that they should be aspirational and inspirational. So Southern should be moonshots. Um, and they are, uh, you know, uh, goals to achieve. And the prevailing cultural norms should be one of empowerment and autonomy. As I said before, both giving empowerment and also taking empowerment in terms of doing them well, it's very easy to do anything badly, including OKR and something that come across quite often is all of the characteristics on the left being wrapped up in a rapper called an OKR.

00:22:12

So you've got chaos, but actually they are just projects. They're projects written as, okay, ours, that they are cascaded top down. Um, they don't update. And with a charitable intent, it's kind of not surprising because people have a limited velocity to unlearn and relearn. So it requires coaching and it requires time and to do okay. I was well to have Oko chaos. They should exhibit the characteristics on the right hand side on this slide. So now let's look at some of the common anti-patterns and patterns of OKR. So let's look at some of the okay. and not okay. Okay. What are the lessons learned? So, first of all, with an, okay, okay. Are some of the patterns, this is an annual objective to be achieved in 2021. And the objective is to have number one market share in Latin America. When we look at the patterns.

00:23:09

So for the mission for the objective, it's an outcome, not an output. It's a very clear outcome, which is we want to be number one in terms of market share in Latin America, it is both inspirational and aspirational in terms of measurement. They are all measurable. One to five, every single one of them is measurable. They're articulated as verb measure from X to Y for example, increase net promoter score from plus 40 to plus 60, they are measures of behavior. So for example, number one, double the ad click-through rate that requires a change in behavior increase word of mouth referrals. Number three from 50,000 to a hundred thousand, that requires a change in behavior grow daily. Digital transactions requires a change in behavior. There are leading and lagging measures. So the first four are leading measures. They're an indicator that you might increase your market.

00:24:06

Share. The lagging indicator is number five, which is you've increased your market share. If you don't have leading indicators again, you're flying blind. You don't really know if you're going to achieve your outcome and you can't exhibit genuine agility because you can't pivot because you haven't learned anything. So you need early and often feedback loops to know if you're on the right track, they measure movement towards the objective. So you should be able to plot the trend. So for any of those, you can plot a trend from the, from number to the two number. They should be measures of value. So for example, customer net promoter score, you know, having happy customers is valuable. Word of mouth referrals is valuable. Growing daily. Digital transactions is valuable and no more than three to five key results per OKR and it's business. And it is one it's the organization, not some business OKR and some it okay. Ours. No, they are combined the organization together achieving business outcomes.

00:25:09

I want to break in for a quick second here. Uh, so one of the points of contention, uh, actually at the conference was, uh, whether product roadmaps should, or shouldn't be part of the OKR process, whether they should show up in the OKR artifacts, Mick made a very strong claim that they should not be because there's a risk that it becomes a tool for leaders to bludgeon their favorite feature onto the teams. Uh, John, you took the opposite stance that, uh, product roadmaps need to be a part of the process. Uh, could you comment on this and, uh, very specifically verbalize your position on this?

00:25:40

Yeah. So I've seen this interesting dysfunction and deployments of OTRs, and I think one of the ones that, that John's pointing out with here is absolutely the case, right? We're just, uh, some managers have been trained on embryos and they basically just reframe embryos as okay. Ours and basically nothing improves. Uh, another interesting dysfunction I've seen is when you've got, again, snapping back to the behavior of just telling people what to do, telling teams, what needs to get done without actually getting them involved in the outcome. And so in that case, they get used and basically create a waterfall roadmap of telling the team what to do and actually without even specifying the outcome, because you actually saying, no, no, we need know we need this feature that feature in this or that application. And so, uh, you know, my conclusion from that was we just, we need both, we need the outcomes and there needs to be as well, hear John say a roadmap of the outcomes. And the roadmap also needs to be first class as well, right then the road that needs to drive those outcomes. So these things are interdependent. Uh, John, I had been having a really interesting discussion how they should actually be connected and practice, but what we do when we encourage others to do and what see succeeding and then technology companies is when both are first class citizens. And they're basically interdependent in the right way,

00:26:51

Uh, specifically, uh, product features. And those we're going to have to do not show up in the quest cascade as, okay, let's go down

00:26:58

That's right. I mean, there'll be thematic things, right? So that, you know, uh, let's say this, I'll just give an example of Tasktop right, where we've got a, you know, we're doing a lot of federal business, so we need to have FedRAMP certification. That's a really big, uh, top-level OKR for the company, but that cascades into roadmaps that are autonomous and that each valleys from sets themselves rather than, you know, saying this and this and that thing gets done. So you do end up with thematic OKR, but of course, then you have to, you know, the old care's not implementing FedRAMP, it's not a feature, which is what, this is like a massive set of hundreds or thousands of features by the way. But it's actually helping federal customers succeed and deploy in moving from project to product, uh, and being able to quantify that in terms of say number of customers. So exactly

00:27:40

Awesome. John, I don't want to verbalize your point and, uh, maybe the, uh, uh, the illumination that, uh, came in a discussion.

00:27:49

So, um, my view, it's very important to have your outcome roadmap super-important and you're out your outcome roadmap is ideally multi-year annual quarterly, and all of that's articulated as outcomes as, as I was just talking about on the talk, it's a, it's alignment, not a cascade, so it's not a hand me down. It's not, here's your team, here's your OKR, but rather here's the, here's the level above OKR. Here's the multi-year intent or the annual intent. Now team, you write your own OKR or, okay. So your OKR roadmap is, is super important to have one because it's your flags that you're planting in the future. They're the, they're the know the aspirational outcome, but without stipulating, how you're going to achieve it. So the, the kind of the anti-pattern that make mentioned just then is if organizations articulate their own chaos as a series of solutions as a product roadmap, you're back to deterministic thinking, you're back to old ways of working, you're back to having a gunshot.

00:29:01

Um, so you absolutely need your outcome roadmap first and foremost. Um, and then depending on your context, you may need to articulate your product roadmap, which is a list of features. And that depends on your context. If you have stakeholders or customers or marketing or a website, when you're saying here's, what's coming next, you might need to do that. If you're in inside the company or, you know, super agile and you might not need to do that. So my, my advice there is, if you don't need to do it, don't do it. If you do, it's kind of just in time. So your kind of feature roadmap, your product is kind of just in time, uh, one learning from one company, what they did, they used to put dates on their features publicly. They changed, and they said now, next later,

00:29:55

Awesome. Now, next later, that's awesome. Uh, and, and, uh, Julia, you actually, you sort of lampooned, uh, that scenario where, uh, OKR is just a degenerate into a set of, uh, features being cascaded down the organization. Uh, uh, I'm assuming that you resonate with the stances made by both John and Mick.

00:30:16

Yeah, very much so. Um, where I worked, I'm not responsible for the government UK roadmap, but you can find it online and they very much use now next later. And they're very much using outcome framing as well, even in public. Um, the thing that I wanted to add though, is I don't think it's new. This isn't something that, okay, all's forces on you. It's having these different views of your roadmap. If you have a public roadmap and at the moment you're putting features on it, you're not putting on your public roadmap. Oh, and we're going to make this change to increase our ad revenue or to make it more likely that a customer is going to subscribe to this particular premium option. So you already have more than one view. There is no such thing as one roadmap that's going to suit all needs. So what is your roadmap? Is there a release plan to people need a release plan? There, there are different roadmaps for different situations. This is not something which, okay, this is going to inconvenience you with it. It's just a different framing.

00:31:07

Brilliant. Okay. Thank you. Let's uh, if, unless there's any objections, let's go back to the video. Awesome. Thank you.

00:31:13

We are taking a look at, uh, natto K OKR. So the objective here is to deliver project Platypus. So the anti-pattern looking at the mission is it's output rather than outcome. Platee what, and this is, um, maybe surprisingly, uh, fairly common where I quite often come across this with organizations where there are projects with names and you might ask teams, you know, what's, what's the benefit of this project. And quite often, people don't know because it's been passed down the chain. Not really sure what the benefit case was when this was first put together 18 months ago. And so no, one's really focused on what the, what the, the desired benefits and the desired outcome is. It's not inspirational and there's no time box on this one. The duration is not clear. Is this an annual objective, a quarterly objective, a multi-year objective in terms of the measurement and the key results.

00:32:08

Number 1, 2 11 is a task list. This is an activity list. These are not key results. They're not measurable. They are not measuring a change in behavior. There are no measures of value. There's no leading indicators of value. There are too many key results, which is common. When organizations start on this journey, it's it only as well. Whereas the business value, even for initiatives, which only involve it, there has to be some business value. Otherwise, why are we spending money doing now onto the mindset? We've looked at the, uh, the mission. We've looked at the measurement now on the mindset and the patterns around the mindset. Um, the background image here is a Japanese Zen garden. And I think it's appropriate for them because okay, ours are as much a philosophy as they are a framework it's culture. Overprocess watch out for new labels on the same old behavior.

00:33:06

Um, you all have not, okay. Okay. Oz. If the approach is to treat them as if they are a process and a framework and leave the culture, unchanged here requires a mindset of emergence, experimentation, and empowerment. Acknowledging that the future is unknowable, maintaining options and optionality, running experiments, learning quickly with empowered teams. They are top down, bottom up and sideways. Less is more, there should be no more than three to five. Okay. As per value stream, there should be a regular cadence of review to reflect and pivot. And again, it's business and it together delivering on a business outcome.

00:33:51

Now taking a look at OKR alignment. So in particular, this is around on and complexity and large organizations. How do OKR is align? First of all, if we do a double click on a quarterly OKR and see what's inside it, how it what's comprised within an OKR, you can break them down into experiments. I like to use the word experiment because the future is unknowable and let's face it. They are experiments. You can equally use the word initiative, or you could use the word epic. When you break that down, you have daily stories. You might have weekly iterations, and ideally you have continuous delivery going up towards strategy. Those quarterly outcomes are aligned to an annual outcome and you'll have multiple value streams. Assuming you have multiple value streams, assuming you have multiple products, you'll have multiple annual outcomes for larger organizations where you've got multiple business units.

00:34:47

You'll then have multi-year okay. Ours are aligned to the business unit. So if this was a bank, the area on the right to might be an investment bank in the area on the left might be a retail bank. And then you've got the group level strategic objectives. So these are the strategic objectives multi-year across all of the business units. If we looked at a worked example, imagine you're a retailer. Um, before we get to that on the left-hand side, it's both top down and bottom up and it's continuous strategy alignment. So you've got a continuous feedback loop on your multi-year strategy. On the right hand side, you've got regular value realization and you've got a feedback loop and you can pivot. So you've got typically a monthly cadence on your quarterly outcomes. So at least once a month, you've a feedback loop on your key results.

00:35:40

And that's it. And that's incredibly powerful to be able to exhibit genuine business agility. So imagine you're a luxury retailer worked example, strategic outcome, top three most valuable luxury brand. When you then go into the business units, luxury bags and fragrances luxury bags, you break that down. The goal is to be top three market share in China for fragrances top five global brand. Break that down within luxury bags, you've got handbags, the handbag value stream. And the goal here is to increase market share in China within the next 12 months, break that down into a quarterly outcome. The goal is to increase markets market share in Shanghai, break that down into experiments. In the first month, we're going to run an online promotion and we're going to hire some social media influencers to take selfies with our bags and post them on Instagram. So that's how you slice the elephant.

00:36:35

That's how you slice it into value and learning. And that's how you align. And you can align across value streams now how to get started or how to get restarted. So to get going, think big, start small and start, start broad and shallow, which is your strategic top level multi-year outcomes. And then narrow and deep. Don't try to big bang it, narrow and deep pick pilot areas. Uh, go down to your quarterly outcomes down to your monthly experiments in your daily stories. I have a bias to action because the only way you will learn is by doing don't spend 12 months in PowerPoint. I point OKR champions. So have some named people who are okay, our champions to provide support, be patient and be resolute, expect to get it wrong, expect four to five quarterly cycles to be able to fully embrace, okay, ours, it's common sense, but it's not common practice.

00:37:32

And like most things it's actually hard to, it takes a lot of unlearning and relearning to do it well, create a community of practice and OKR community of practice for shared learning, expect the reinvention of the PMO. So this is, uh, the PMO reinventing itself to the VRO, the valuation office to support the organization on OKR. If you're interested in more on that, I did the talk in June, 2018 at the DevOps enterprise summit, which, uh, where, where we have a bit of a case study on that and experiment and enjoy the journey, uh, because it's about the journey rather than the destination in terms of the help I'm looking for. Uh, I'd love to hear your stories, your adventures with OKR. I'd love to hear about your patterns and your anti-patterns. Thank you very much.

00:38:22

Uh, just a brief, uh, so John you've presented a wonderful litany of the not okay. Not okay. Okay. Ours. I just want to briefly ask the panelists to what extent have they seen, uh, you know, elements of that catalog of anti-patterns, uh, in your, in your work, uh, and you can safely, uh, or have you seen, has your friend seen those anti-patterns at work? Uh, Julia wants that with you. Okay.

00:38:50

Um, I'm quite lucky in that at the moment I'm working in an organization that has been agile from its inception and has always looked at user centricity and outcomes from its birth. So, um, we have a really strong culture for that. So it's actually not a case of learning a different way of working. Probably the thing that's been most difficult for me is picking my battles. I think sometimes, certainly when I was working in, um, dev ops teams, looking at for infrastructure, if the task was, um, get off of a particular technology, because it's going end of life, we know we're doing it to reduce risk. We could create a risk framework and start scoring things in order to give us something that we can measure and turn it into an OKR. But we don't have that right now. And actually is this the hill I want to die on this week. And so I was finding myself having to think, how can I compromise this on actually for pragmatism, I'm probably going to have to, um, whereas in other places it was much, much easier to draw the line to outcomes we could measure easier. And I realized it's not about measuring what's easy, but sometimes if the measure doesn't exist, inventing a whole new measure, isn't the thing to do right now.

00:40:04

Awesome. And make, um, uh, and by the way, uh, the UK GDS is such a remarkable organization for so many reasons that I'm learning so much about, like why it is so remarkable and trying to understand how they can preserve, preserve such greatness over years. Uh, so we'll learn more about that later, Mick. Um, so over our many calls, uh, I think it's safe to say that you have seen many instances of these sort of anti-patterns OKR processes going wildly wrong.

00:40:32

Uh, yeah, absolutely. In terms of, you know, working with organizations, I, I see it all the time. Um, I saw some yesterday working with fortune 50 health care company in terms of, I just always ask the way that peoples up there are trees and cascades and, uh, and it's, it's just, it's a fascinating thing now, gee, I hate to admit this, but, uh, I see, you know, within our own organization, uh, I see this every single quarter, right? It's that these, we end up with these task lists and these activity lists. So even, even organizations who've been doing this for years and years, you still end up with this. And I think the key thing is it's, it just takes, I guess the key question is why is it hard? Why do we snap back to these task lists and these activity lists and, uh, you know, the, the key reasons I see both in our customers and in our organizations, for example, if something's not measured is not yet easy to measure.

00:41:21

So the, you know, John mentioned the from, and to, if you don't have a baseline, what do you do? Uh, and I think there's, there's key there's ways to do this. So when we don't have a baseline for something, because we're, you know, something is not instrumented yet, it's not easy to get the data or it takes a whole team to get the data. Uh, do we just say, okay, targeted percentage improvement. Even if you don't have the baseline, you can just just guess improve 5% and 10% reduce your, you know, reduce your flow time by that. So that's, uh, that's one thing. The other thing is that, especially for the quarterly ones, uh, oftentimes the feedback loops structuring things just it's like structured small batch sizes can be very hard to structure, uh, your objectives in a way where you get feedback within a quarter and not make them an activities. So if you want to launch a new learning program, there's a set of activities that you need to do. Will those activities be measurable within that quarter? Or we'll take you the next quarter to actually get, uh, get an outcome such as more people trained or certified or something of that sort. So I think we have to recognize, I think that th th the principles Johns Weinstein is exactly what we need to do and that we need to recognize when it's hard and, and how to overcome it.

00:42:25

Ah, that's interesting. And that is a very curious question, right? I mean, for decades, we've been saying, don't tell us exactly what to do, tell us what you want to achieve. And yet, uh, given the freedom through the OKR process, we revert back to a, I wouldn't say order taking behaviors, but, uh, create

00:42:42

It's amazing. We refer to, this is ordinary human behaviors. That's the,

00:42:46

Uh, by generating the activities, we turn them on. We inflict on ourselves order taking behaviors. Fascinating. All right, let's go to the second video. And by the way, I see this a phenomenal amount discussion. Uh, Nick and Jim are trying to marshal, uh, questions to get people, to get questions, to ask live. I think the instructions were, if you're interested in asking a question to the panelists, raise your hand and, uh, we'll figure out how to, uh, get you integrated in this conversation. All right. So in the meantime, I'm going to start the second video. And by the way, I'm using this feature that we just discovered inside of zoom, we can actually share a video file. So I guess it's being streamed directly. So it's so much better than actually sharing your desktop and playing a video. So that is what I'm doing now. So we can go to mix presentation and let's start.

00:43:36

Thank you, Admiral Richardson. Okay. The next speaker is someone who should be familiar to most of us in this community. Dr. Mccarsten wrote the amazing book project to product three years ago, and as summarizes his 20-year journey, understanding how to truly unlock developer productivity. And so over the years, he's taught us how to properly diagnose flow problems in the technology value stream so that technology can help best achieve our business goals and how organizations can move from the world of project management to product management. Throughout this past year, we've had numerous conversations about how so many organizations are having trouble rolling out or objectives and key results, especially when top leaders overreach into the planning process using OKR is to bludgeon their favorite project or feature into the plan. Dr. Kristen will talk about his seven years of experience OKR. And what do you see seen go right and wrong in large complex organizations? Here's Mick,

00:44:40

Hello everyone. I'm coming to you here from Vancouver and thrilled to be participating in the DevOps enterprise summit. My favorite conference of the year, both in London and in Las Vegas, and the topic for today is OKR and dev ops. How we can go from micromanagement misery to finding flow. So I've now had eight years of experience managing my own company with OKR is I've noticed that over the past, especially the past few months, but really the past couple of years as DevOps has become a CEO and executive suite and a boardroom topic. And as the shift from project to product has really helped more organizations focus on technology and understanding how to start innovating, how to start measuring and tracking for value. Uh, this is, this has become a really key topic where some organizations are using ocairus very effectively to better understand how to connect technology and business and deliver more for their customers, whereas others are failing.

00:45:31

And what to me were, were very surprising ways. So let me quickly talk to you about the, what I've learned within my company, supporting other organizations who are using this methodology and then how you can apply some of these lessons to your own initiatives. So, first of all, objectives and key results are just one way of tracking plans and goals. And some of these concepts will apply to any other ways that you're measuring value that OGs, GSMs, and other approaches, but we're going to dig in specifically to these OKR. So the objectives are the what and these inform actions. And they're really about finding a way of measuring value to how much we're delivering for customers, our market, our colleagues, and the KRS. The key results are the benchmarks for how we're doing that. So the point of OKR is that you have around three to five of those that the cell phone, they factor in how they cascade and how they get and how they spread across the organization.

00:46:20

And they're really, uh, uh, summarize very well by John door's book measure. What matters came out in 2018, I'd actually been applying them for years. It was very helpful for me to see this book. I had my whole leadership team read it, but I found that with this book, that was a really significant gap. And I think this gap is where a lot of the deployments of OTRs fall into. And that's the, how, how do we measure value for software delivery? Because if we don't have a consistent and clear way of measuring value, then what happens? We we'd start measuring the wrong things. If we measure the wrong things, we often get into these pitfalls of actually slowing down our teams of, of inspecting at the wrong levels of cascading, far too low mocking and micromanaging where we shouldn't be. And so in project, the product, uh, introduced the full framework where the full frame was really meant to capture a way of measuring flow and connecting that to these business results, that, that top right part of the flow framework, where flow metrics we've been using for many years now for eight years now, with Tasktop for measuring key results related to flow with how we're delivering value for our customers.

00:47:20

Uh, and then combining those with the business objectives of, uh, around the other financial metrics and all the other metrics in organization. So I'll show you how we can combine these things and have them work effectively, and how in the end it really is all about making sure that you include flow metrics and your key results in order to connect business and technology. So, first of all, the question of why flow, I think we in, uh, in recent news, we've had really interesting examples of why flow is so important, how problematic bottlenecks can be. So we saw the entire shipping industry grind to a halt. Then our products all get delayed, uh, in terms of getting the, getting to getting to consumers because the Suez canal became a bottleneck. And if we look at this and we actually realize that when we have this kind of bottleneck, the entire system slows down, there've been these amazing visualizations that show how it's not, it wasn't just the ships around the canal, but the global shipping system ground to a halt.

00:48:13

And yet we're seeing these kinds of things all over organizations. And the question is, are we actually seeing them and improving them or not? So I think if we take as leaders of teams, of organizations, of value streams, if we take our job as improving flow, adding really focusing in on that second ideal of focus, flow, enjoy for all our staff, all our colleagues and for what we deliver to our customers. We need to know where our bottlenecks are. Uh, as, as gene Kim, uh, channeled gold draft and the Phoenix project, any improvement made anywhere besides the bottleneck is actually an illusion. So if we're investing elsewhere, other than the bottleneck, we're not helping flow, we're not helping that focus or that joy. And the thing that made me so excited about OKR is, is that there, that I found them as a really effective tool for speeding up flow for changing organizational conditions, investing in bottlenecks.

00:49:00

But what's, I've also found fascinating is a lot of my colleagues and large organizations we've been deploying. Okay. Ours have been finding is that the okay is actually slowing them down. And so this, this seems horribly ironic and something that we really need to fix. So let's look at how these things have gone wrong. And I've had a lot of conversations that a lot of organizations have them have they been deployed these okay, ours and having things go sideways. And so these are something the key problematic things, themes that I've seen, first of all, okay, ours are being used for micromanaging teams and value streams. So rather than setting these organizations, directions and paths, and really helping everyone navigate towards that common goal, they're actually being used to track specific features, muck with roadmaps and reprioritize things for people. Uh, and they really only, I didn't really think they were meant for that because in the end, when, if that's what's happening, we're actually re-introducing waterfall planning into our dev ops transformations and our agile processes, which is again, kind of antithetical to the goal of why we're all here.

00:50:01

Uh, another really fast thing thing is using the business and financial metrics as the only key results. So what will often happen is that financial metrics such as a cost metric, uh, or a, or a revenue metric could be quite divorced from what a team is doing. You might have a platform team creating the new cloud infrastructure ramping up a whole lot of SRS who might not map directly into a revenue metric, but is still going to help this organization succeed and eventually drive to faster revenues and, and more of the UN and more market leadership. Another key pitfall that we'll go into is using only team proxy metrics as the, as the key results, that being all that we measure. So teams have a lot of detailed metrics, but a lot of different KPIs that they need to use. But if proxy metrics, they're not metrics of value of flow by very specific, the team's work that can really derail OKR efforts. Uh, one of the biggest one it moment. And the key thing is that if these, these practices of using ocairus for something they weren't intended for, for measuring, for not measuring value result and conditions that make OTRs work against you, they make it really make you feel like things are slowing down, which means chances are they are slowing you and your teams down. So let's look at what bad OKR is, where they come from, what they do. So again, if we look at a specific value stream, you can see this value stream,

00:51:13

Actually, why don't we stop for a quick moment? We have a question from Catherine, um, at see, can we figure out how to get Catherine on with the question first time? Hi, Catherine.

00:51:31

Hello? Can you hear me? Yes. All right. My screen is frozen, so I don't know if I'll be joining you on video, but thank you so much. Um, a quick question that I get all the time is we're looking at flow framework. We have our business results going to be business outcomes, and we may have various levels like in a safe hierarchy. And I know John, you had a slide on, uh, OKR is at various levels. Um, just curious to get thoughts on one, you know, is it okay to have ocariz at various levels? Like in the safe framework, we might have the portfolio solution program team. Um, and also I know the answer is going to be about the feedback loop, like the cadence to review them up. So not only may you have a different levels, what cadence does that look like? So two questions in one.

00:52:21

Okay, awesome. And by the way, before you, uh, we hear from the other panelists, um, Catherine, could you just briefly introduce yourself organization role?

00:52:28

Yeah. Currently I'm at Tasktop as a principal flow advisor and I came here because I have been through this pain in large organizations and helping coach through this shift from output. So yes, super excited to have recently joined task squad,

00:52:49

Anyone.

00:52:51

So Kate, you asked about the, the nested OKR that I had on mine. Um, personally, I believe that to be really important because that gives you your multi-year intent, your annual intent and your quarterly intent. So you're slicing the elephant carpaccio rather than eat the elephant, whole thinner slices of outcomes. And it's, it's really like your kind of breadcrumb trail to your north star. Um, so I personally think it's really important to do that.

00:53:29

I'm going to yes, and John, which is, I agree it is important. And sometimes when you're halfway down an organization or near the bottom of an organization, if those things haven't been done yet, you can still create them at whatever level you're at. Um, the, where it would be broken is if you have them high up in the organization and they weren't actually happening anywhere that any delivery was happening. Um, but yeah, the higher up the organization, the better, but don't let a top level set of chaos, deter you from seeing them at whatever level you are at. It can be a way of getting, um, getting a strategy articulated if it's not very well articulated, or it's not very, if it's not being articulated in a way that you can make directly relevant to the work of your team, by coming up with what you think the kales are, and then showing them to the people above you in the chain, that's a way to see whether you've understood their intent.

00:54:26

Brilliant. Yeah, I completely agree. And I think if you think back to John's example, I think what's so important and it's, it's similar to what Julia is saying here as well, is that at each level you have autonomy for setting them right at the top level, you've got the strategic one year and three, or ideally, uh, at the next level down, whether it's a department that value stream a program, uh, you need the autonomy for setting them in terms of how they will drive the organizational ones, but then both at the quarterly and annual level. And I think that's some of the words it's very easy for some of these work item hierarchies to just quickly degrade into requirements to these very deep requirements, threes. Okay. There's an opportunity to stop that right, to make sure that at every level, whatever work is being planned is connected to the outcome that was set by the people at that level who understand the problem and how to, how to drive for the outcome. The best context is always important.

00:55:16

Uh, John, can I, and if I could, yeah, just don't just to build on Mick's point there, uh, Kate, in your question, you use, you said you mentioned safe personally. I think what I had on my slide is different. This is not about, uh, portfolios and projects and programs. I'm talking about proper nesting outcomes all the way, uh, which I think is different. Um, so just wanted to add that thought. Um, and also just to add one other thought is watch out for orphaned. Okay. Ours. Um, so what I, what I've done in the past is I've run. Um, I've run a nightly report, which reports on anything at any level. It doesn't have a parent apart from the top level, because if you don't have a parent, you might be going off on a tangent, you might be not aligned to strategy. So this actually becomes a portfolio control in your control environment, which if you're regulated, regulators are interested in this. So to actually make sure nothing, doesn't have a parent, everything has a parent.

00:56:28

Awesome. Thank you, Catherine. All right, let's go back to the video and we're nearing the halfway mark of past the halfway mark. So, uh, if you are interested in asking a question indicate, so by raising your hand in the, uh, uh, in the zoom channel, I think that's the latest instructions. All right. Thank you.

00:56:44

It has a pretty significant bottleneck in it. A lot of features and other flow items are trying to get through, uh, but they're not getting through. And what will often happen is leadership will actually be using okay, ours to micromanage. What gets through to mark with the roadmaps to prioritize features, to ask where's my feature as that's happening. And they all care it's being used to manage deliverables. Uh, the bottleneck is not visible because everyone's just wanting more features more quickly. Uh, and we're not seeing these dynamics of flow. And of course, if these OTRs are coming top down, being pushed on the value streams, what's happening to our value streams is that, uh, that their capacity is being overloaded. There's more and more can progress the flow load increases and even less makes it to the customer. Even less value is delivered. Contrast that with good, okay, ours, which should not touch any of what's being prioritized.

00:57:32

We've got protestations frameworks, agile frameworks for that, but instead those old care's focused on, on reducing that bottleneck, removing the impediments and adding value to for more quickly providing more focus on flow. So these, okay, ours are actually about looking at how we get more value through and remove those wait states, those impediments and surface those bottlenecks because sometimes the bottlenecks will be within the value stream. There's a lack of some continuous integration, deployment automation, some overly slow security reviews. Sometimes they'll actually span value streams. So, okay. That's could surface those bottlenecks. If multiple value streams have dependencies on say one platform or lack of API APIs. And of course the key thing is that they need to be, they can help balance with the features that we need to deliver. So, okay, guys can have balanced roadmaps because they allow room for improvement, which, which the key results can drive while of course, roadmaps are both delivering more value to the customer more quickly.

00:58:25

So the key thing to understand this is that we've got different planning dynamics that we need to keep separate, that we need to keep separated. So we've got roadmaps and we've got plans, and those really define what gets delivered and in what order and how those things are prioritized. So that's analogous to the order of the container ships, trying to get through that Suez canal. We have value stream specific OTRs, and this is where the valuations themselves, this team of teams construct can accelerate the flow and can surface these bottlenecks. For example, if there's a shortage of staff in one area, a lack of automation, some other process impediment, this is really the value streams themselves, trying to wine that canal and need the autonomy to do so. No one understands better how much investments should be made in automation and tech that reduction and process improvement.

00:59:07

Then the people working on that value stream themselves. As you know, as, as you've heard in the other regions and say, we need to delegate, we need radical delegation to the value streams to set their own, uh, okay. Ours. And then we do have the organizations that company-wide OKR, the one specific to a, to a large line of business, and really they're the, okay, there's an opportunity to create the conditions and organizational structures to enable flow. For example, if we're, we've got a lack of investment in a common cloud delivery platform where automation on that front, these are an opportunity to accelerate that. And this is really, uh, analogous to building a new canal, to making sure that we're not reliant on a single canal and totally changing the conditions to enable faster flow across the entire organization. So to understand how we do that, we really need to understand the different types of metrics that we're going to measure it because each of these OTRs and need to use different metrics.

00:59:58

So the business metrics are generally well understood. Those are costs. Those are pipeline conversions. Those are retention rates. Those are profitability numbers, uh, but usually they're a lagging indicator of technology work. If we deliver these features, those tend not to turn into revenue overnight. The flow metrics help tremendously here because they track the improvement. They track our ability to deliver more value, to delight our users with even more of the features that they're looking for. And so they're a leading indicator of value delivery that then turns into a business metric and very importantly, team metrics. The telemetry for four teams need to actually be specific to the teams and not conflated with the OTRs teams need a lot of telemetry to do their own decision making that this isn't making does not need to percolate up the organizational levels. It needs to allow the teams to move very quickly, respond very quickly to incidents, to outages and to other needs.

01:00:46

And again, be a completely separate set of telemetry rather than the ones that support organizational improvement such as the business level. Okay, ours. So let's just quickly go through an example. And this is an example as a, as an insurance organization, uh, where that's deployed OKR is would actually using some little data sets here. And some exemplary OKR is where what they want to do is become the most innovative instrument industry. This is great because the objective is actually linked to their core values and innovation for their customers. They want to see that turn into 30% market growth at 50% reduction in the time that it takes the provision insurance policy. So that's really important because that OKR cascades down to how prioritization is done. If a roadmap item is going to make it easier to move customers faster, remove calls or places where things are being dropped, we'll know that we're on track on this of this OKR.

01:01:33

And then the key thing is these two, okay, ours are balanced with a flow efficiency improvement. So flow efficiency is the ratio of wait states to work states. This actually helps fuel that, that third ideal from the unicorn project around around improvement of daily work, because it allows everyone to take time in support of this OKR to actually improve how quickly they can get value for customers and drive that efficiency. But let's take a look at how the platform value stream, which really supports all of these new applications. All of this new innovation actually interpreted those OKR hours and turn them into some very significant change for the organization. So what had happened is in terms of moving fast and want to grab more market share, there was a whole lot of technical debt and crude and the shift to cloud and the shift of all the different customer views, the policies, uh, and all of that infrastructure as a result, the costs had ballooned.

01:02:25

So there was some lifting and shifting more than anyone wants to admit that was going on. And the, it was clear that it was no longer scalable to keep building these applications. In this way, the team themselves realized what had happened because they were not making efficient use of storage services. They realized that there were much better ways of they could just focus in on technical debt. So they actually used and set their own OKR saying, if we focus in on technical debt, we can dramatically change this hosting profile and have finance stop worrying about this. And everyone realized we really can scale the 30% market leadership as our goal. They did that over the course of three months of investment technical debt. They dramatically reduced this hosting cost bubble. They reduce hosting costs by 75% and they had proven that this can be done and that the entire portfolio can actually be hosted in this modern way.

01:03:14

Now let's take a look at the actual, uh, customer facing applications. So this is the policy value stream where really the focus was making products that these customers love. There's a lot of competition from insurtech organizations. And so a 20% improvement in NPS was really a key part of hitting that objective objective that was set by this value stream itself, the ones producing all of the mobile and web experiences. And to do that, it was really what's about delivering those features that delighted customers, more better authentication experiences, making sure that they can make it through the whole, the entire provision process quickly and with some joy. So thankfully the organization already had a flow efficiency from switching to the experience experiments on their way. So they knew where to target the process improvements. It turned out that the bottleneck and there's has to be only one at a time, uh, was in verification, that business was needed.

01:04:01

Verify all those features, making it, the customers. They already had good automation of feature delivery in place, but all these weights states came from waiting on the business and all the meetings that were happening. So the team targeted and was very public in their own, okay. Yards around this zero days, wait, state on business input for shipping. Those features once that happened, flow time was cut dramatically. We can see it go down from almost 30 days to averaging. Under 10 days, new features were being delivered to customers, mobile web, and under 10 days for this organization, from that full-time reduction, we actually did see the net promoter score, how happy the customers were with the applications rise, of course, as a lagging indicator, because it took time for them to start adopting using this features. And this really helped that company on that objective of cutting the time to provision policies in half, because they have happy engaged customers who are getting to successful policies.

01:04:50

So the whole point here is as your plan yoke is OKR is as you're looking at rolling them out, make sure that you're measuring the flow of value and the flow, the flow of value by that, that fifth and most important ideal of customer centricity. It really is all around what you're delivering to the customer to do that useful metrics as the key results to remove those impediments to the team, because fundamentally the teams know how to deliver that value, do some radical empowerment of the value streams to set their own KRS and them to deliver those business objectives, remove the bottlenecks and signal to the organization as a whole. If there are these systemic bottlenecks, slowing them down, such as missing platform, API capabilities, rather infrastructure, let the teams use their own metrics. The value streams use their own roadmaps and decouple these things completely.

01:05:32

Okay. Is it really for that organization learning and improvement, uh, and should be kept separate from the , which are really this north star in the end, the organizations live and die, but the agility, the quality and the speed at which we deliver software and okay. There's a great tool for accelerating that. So in terms of help that I'm looking for, uh, I would love to hear more about your experiences in terms of what's gone wrong with the OKR it's. I gave some examples of the things that I've learned. Again, I think they're an extremely effective tool if you're using them successfully, uh, please share this out on the DevOps enterprise summit slack, and I look forward to learning more on the reporting on the next set of, of experiences on how to accelerate flow through. Okay, ours.

01:06:16

Awesome. Thank you, Mick. So, um, to the panel, uh, mic made the, I think a very compelling and powerful claim that improvement of daily work, uh, should, or even must be a part of the top level OKR is. And I, I think that, uh, insurance claim processing example was so great, right? That resulted in massive efficiencies in terms of time to market, uh, John, uh, you didn't make that specific claim. Uh, can you talk about, uh, that omission? Uh, and to what degree do you believe that the improvement of daily work, uh, has to be a part of, uh, top of ours?

01:06:49

So I've seen it both be part of the top level. Okay. Chaos, and I've seen it also successfully not be at the top level. And what I mean by that is quite often for, for large organizations, top level outcomes, typically are things such as growth return on equity is, you know, in a market economy, this is all about, are we generating a return on the equity deployed in the company? So that's the obvious one. Um, typically there might be one around customer obsession. Uh, there might be another one around, for example, simplification, uh, which is kind of like the kind of cost reduction, the efficiency one. So you've kind of got effectiveness. You've got efficiency, you've got customer obsession. What I've also seen work in terms of flow, ways of working not being at the top top level is actually the next level down. You might have one around ways of working around flow because that in itself improve it as Mick was talking about flow itself as a leading indicator to the fact that you'll have customer obsession and you, you need to have simplification. And so in order to have better flow, you have to simplify, you have to remove impediments. So you can almost have it as the neck, as a second level down, which is in order to grow in order to have happy customers in order to be more efficient, we need to work on flow on ways of working or better value, sooner, safer, happier on the flight on the slow framework,

01:08:30

Right? And by the way, just to be very specific in our call, you said, this is the top level, um, uh, objectives at a, at the top level leadership was customer satisfaction, customer obsessed, radical simplification, and growth.

01:08:45

Uh, those are fairly common kind of top level outcomes. Yes.

01:08:49

Yeah. Brilliant. Um, and, uh, you also, in fact, this is, uh, I haven't actually heard this quote before time to value is actually more important than value itself, just to say briefly, uh, where that comes from and, uh, why it's so important. Yeah.

01:09:07

The Douglas Hubbard who wrote how to measure anything, he looked at thousands of investment decisions made in product development and the single piece of information that he found to be the most useful information for making an investment decision was how quickly will the product be used, not the cost, not what you're building, not who's on the team. Uh, the most valuable piece of information is how quickly will the thing you're building be used, because if it's a really long time, your expected opportunity loss, the chance of being wrong times, the probability of being wrong is very high. So you end up writing off all of your investment, you know, if it takes you two years to actually get any feedback. So the most important piece of information is time to value

01:10:00

Steve sphere. If you're still on, uh, this will be top of our next call, time to value, uh, blunts, uh, the equation of, uh, the, probably of a probability of being wrong times, the cost of being wrong. I think that the unassailable, uh, Julia, um, uh, what is your opinion on the improvement of daily work, being a part of the, uh, top level objectives, et cetera,

01:10:27

I'm kind of agnostic on it, to be honest, I think it's, is it going to happen if it's not, if you have people invested throughout the organization, that it will happen, if it's not, then you will, as long as you're measuring it somewhere. Um, and as long as you have people empowered to make decisions that prioritize it, you probably fine. If that's not happening throughout your organization, if that's not trickling down, then may be, I would maybe something, some kind of measure of the top of the organization would help drive that. Uh, but yeah, as always, I'm a pragmatist, if it's happening and, and if people are incentivized to prioritize it, then

01:11:13

That's the main thing. Oh, I'd love that. In my interview of Admiral John Richardson, former chief of Naval operations, he said, uh, you can tell the, uh, personality of a ship based on who's leading it. And, uh, your comment just reminds me of that, this, that, uh, if you have a leader who is obsessed with developer productivity, then it'll be very natural that the developer productivity measures will be part of the, uh, top level OKR is Mick.

01:11:36

Yeah. So, uh, I'll touch on this one because I learned some interesting things from a, I've been working with Philippe, Casper, who was one of these sort of business on a product oriented OTR gurus. And when the main points that he makes, which was really just back to the, you know, to the principles of dev ops is that we can't okay. Can't be around building wedding cakes, right. You'd have to build cupcakes and then measure it, measure the cupcakes. Uh, and then the key thing is if, if it takes you more than 90 days to build a cupcake. So if it takes you more than quarter to get feedback, you can't use OKR. They just, because it's, you're not getting the feedback. That's again, you're flying blind, you're filming work on people, no sense of capacity, no sense of whether it actually drove the outcome or not.

01:12:19

So the thing that I'm seeing is that, you know, what would we see as high performing organizations? Their, their, their feedback loops are within the sprint that's two or three weeks, right? Um, and then by the time that you're doing your release planning, your business, planning, your quarterly business reviews, whatever they are, where you, where you set your okay, ours at the nine day mark, you've actually learned a lot in the course of that quarter. So you're actually all of that is feeding back into the OKR. So what my realization through all of this is if you've got any organizations, what we're seeding that time to value that, that flow time be more than 90 days when you first, okay, I was going to be shortening it to be less than 90 days, or, or again, you, you won't pray that feedback cycle. So to me, this is just to say is just the same way.

01:12:56

Again, we just, we need to measure, work and measure the outcomes that, that we're creating, but also measure the improvement of daily work because so many organizations are taking too long to drive that. And again, we get into this challenge of if we're only measuring the business key results, the feedback loop for the teams is too slow for them to understand, are they actually driving that outcome? And we need both, right? We need, the teams are doing activities day to day, and those activities are divorced from outcomes in terms of timeframes, right? They're, they're separated where, you know, whether it's a net promoter score or a revenue or market share outcome, or that the work I do today might, I might not have a feedback of three to six months to whether it actually drove that. So we just need to put in place these key results that give us a faster feedback loop.

01:13:39

And that's where measuring flow and improvement you'll work can really help. And of course it would help in terms of actually getting the business outcome feedback faster. And then introducing the ways that we have our people actually actively using the things that we build is a driving net, promoter score, product scores up and so on. So yeah, I think bottom line is I think it's very effective just to measure the improvement of daily work. And of course, and this is back to choice point is that, you know, balancing these things, because I think we need to measure business outcomes. We might need to measure improvement for daily work outcomes, and then we need to measure, are we actually in the process of removing the 12th from our team? So for us, another key OKR that I always encourage others to do as well is to have a employee net promoter scores and employee engagement as a top level, um, uh, okay. Or that you track, because then you'll be balancing these things. If you're working, people do hard to drive outcomes, they'll actually get less productive, less happy and leave overtime as well. So,

01:14:28

Oh, uh, this is great. In fact, I know just, uh, uh, I, ain't going to be listening to this, uh, myself later because, uh, the information and learnings is outstripping my ability to actually, uh, uh, fully internalize. So I think I thought I had one last question, uh, for the, um, the panel, but I seem to have lost track of it. So I'm going to move to, uh, the audience questions. And, uh, this is where, uh, how we'll spend the remaining, uh, 20 minutes before I'll turn it over to, uh, to James. So the first question that, uh, Nick and, uh, James have picked out is from Troy, um, McInnis. And I've lost those questions out here who are, uh, for several years, we've been talking about OTRs versus milestone-based delivery in the panel's view, how was the world, um, how has the world matured towards OKR is over the last five years? Where are we in the maturity curve? And what does it look like by industry? For example, uh, in other words, is financial services 30% or 60% of the way there. That was alright. And that was actually not from Troy. That was from Mikael.

01:15:34

Yes. I mean, I can speak to this anecdotally of kind of the last a hundred organizations have, have been involved in, in terms of their transformations. I'd say about half or more are deploying OKR is, but it's all within the last 18 months that they've started. And I think it's just somehow I think measure what matters catalyzed all of this somehow 20 18, 20 19. So I think it's, it's, it's quite new, which is why it's, it's kind of exciting and a good opportunities though.

01:16:04

Yeah, I'm just about to it. I think there are Silicon valley, west coast companies where John Doerr has, has had a hand in them. I've been doing this for some time, like Google, uh, LinkedIn, Uber, Facebook, and so on. And, um, in terms of the Shire horses, rather than the unicorns, they're not by degree with what makes that, yeah, this is it's, it's still early, fairly early on the journey. I think there are some organizations that have been doing something like OKR. So the talk that I gave in 2018, you know, from my previous role, we, we were doing something similar with that structure. And I think it's now evolved to, and we didn't actually call them OKR because that was a term that nobody knew and no one understood. And they'd be like, what are you talking about? But now it's got to that tipping point where it's common, common language,

01:16:59

Julia, any, uh,

01:17:02

Not being a consultant. Um, I've worked in only a handful of workplaces over the last 10 years. So don't really feel like I've not seen enough to be able to comment all I can say, the organizations I've worked across isn't universally. Um, probably the thing that I would say is having worked in mostly large organizations, um, you can get pockets of practice where the whole organization isn't, and this is, this is why I talk a lot about actually, do you know what, even if you don't have this north star metric coming right down from the top, you can still do things in sub organizations within the can still be meaningful and impactful where you are. They won't be as valuable as if you were connected right up to the top, but they're not worthless. Um, so I guess that's my observation is you get different levels of maturity in different places within logs organizations, which in itself says something about them, the charity.

01:17:56

Yeah. And by the way, one of the big insights for me was a boy, like a chaos can be done at any level and should be done at any level. Right. Because we really do want to steer towards value. Okay. So here's the trust the question from Troy, what is the OTRs don't move when and how should it be? It they'd be abandoned. I'm a very curious about the answers, because I think prioritization typically and focus tend to be a good thing. Uh, but yeah. What if, okay, or don't move, when is deleted, do you capitulate and, uh, uh, you know, uh, toss it out. Um, anyone who has a strong opinion,

01:18:33

Sorry, go ahead.

01:18:34

I was going to say two important things to determine whether the OKR is moving or not. Number one, make sure you have leading indicators as well as lagging indicators, because if you don't have leading measures of value, you're not going to know whether your experiments and your activity are having any impact or not. And whether the lagging indicators are going to move eventually. So examples of leading indicators might be more clicks on the online advertising, more customer inquiries, more, more, uh, product inquiries, more registrations, you know, more newsletter subscribers. Um, those might be leading indicators to the fact that you might have more customers, but you might increase your market share. So number one, really important to focus on the leading indicators, because that gives you business. Agility enables you to pivot thought, number one, thought number two, back to the time to value, you've got to have regular slight, you know, really regular small slices of value because then you can learn and you can pivot to about time to learning time to value. Then you can pivot. So to Troy's point, if you've run a whole load of experiments and you've pivoted a whole load of times and still nothing, you know, the leading indicators, not budging maybe now is a good time to kill that. Okay, I'll pull another one.

01:19:51

The dependent variable may not be connected at all to the independent variable, Nick, Julia.

01:19:57

Well, I think that, that's the, that's the key thing is like, when, when you're measuring, you'd always said you have information at the end, right. And maybe it was the wrong metric. Right. Which is why, again, I know every quarter evaluate whether that was an effective measure or not. So, you know, take a day, they come in the measures for four products, right? Like daily active users, uh, if it's not moving, is it that? So let's say it stayed completely flat. Even though we did all these activities over the course of the port and delivered all this, all this good stuff, there's information. And the fact that it stayed flat, just like there would be an inflation effect that potentially went down and maybe it's because there's a change in the market and all this and people are actually have gone to the competition. And so we've done all these great things, but people now are using a competitive product, or maybe we built the wrong things.

01:20:38

Maybe we had an issue of designer or these experiences is not as good as we thought it was. So I think that the key thing is when, when you're evaluating, if you're evaluating both the sort of intended, uh, outcome, and whether we hit that and if not, why not? And if we blew it out of the water, you know, why was it something we did there wasn't actually that, uh, I, I I'll give an example of it there. I though one of my colleagues who runs a payment processing company, so basically his, his, okay, as Bitcoin price goes up, his OKR is go up and as it compresses go down has, okay, let's go down some, the top level company wants because, and that has nothing to do with what the teams are doing. It's just because there are more transactions happening that they're authenticating, uh, around Bitcoin. So it took a while to figure that out that it wasn't actually the internal actions talking about it was fluctuation of the market. So then that's the key thing is every quarter we get a chance to learn and both question, question, you know, why the number moved, why it didn't move, or whether it was the right thing that we were measuring. And that's, what's so great. Another thing that's great about OKR is it creates a simple process on evaluating how we measure and whether it's, we're measuring the right things.

01:21:46

Julia, um, I think also bear in mind, to some extent, all of this is a gamble. All the work you're doing, there is no certainty around this. You are gambling with the resources of your company or your state or your shareholders, or in my case, public money, um, in something which you think is gonna achieve an outcome. And some bets are pretty near a sure thing, but still not enough that you want to take your eye off the ball. And some are much Wilder a gamble. And it's, it's a little bit like going to the casino. If you don't know how much you're prepared to bet on these things, you're going to end up going broke. Um, so there may be some things which are going to be so amazing if they pay off, but actually you'll be prepared to, you know, well, let's give this another, this, give this another month and see what the numbers are doing in another month.

01:22:31

But I try to get teams to look at, to decide beforehand, how many months would we invest in this thing? Because you will always that there is this massive temptation that comes with exception bias, which is you'll get a month in your numbers. Aren't doing what you said you wanted them to do and you'll go, oh yeah. But that was because it rained that weekend or that was because Bitcoin's value went down or that's, that's because of the thing. And it may be true, but there will always be something. And that may have been truly exceptional, but it's really easy to kid ourselves. Oh, no, no. It was just because it rains, it rains all the time. It's just, if your thing isn't successful, if it's raining, it's probably not successful. Um, so yeah, being as honest with yourself upfront and writing some of these things down, so you can hold yourself to account in future is a really useful thing to be able to do.

01:23:21

Um, yeah. I, I remember saying to a colleague years ago, um, past me is an idiot and I know this because I can see some of the things past me did and they don't make any sense. Future me on that basis is probably going to be an idiot as well. Currently in the present, I am completely rational in this moment and I make wise and sound decisions. So right now I'm going to write some things down to hold future me, who I know is an idiot to account. And I've found that's been really useful in trying to control for biases with myself and with teams. And, you know, he usually raises a smile as well.

01:23:54

Yeah. Probably way it occurs to me. This actually gets to a question that Jackman son asks us, like, do customers really care about market share or whatever. Uh, but I think the mark of a great OTR is that eventually someone gets fired if you can't achieve that objective, right? Whether it's revenue growth return on equity, at some point, if those are sufficiently dismal, right. Someone gets fired, right. Potentially the person at the top. So I think that is a great mark of a great OKR, right? At the top level OKR, is that, is it a important enough where someone's actually betting their job on it? You know, I have to, uh, there's a question that came in that, um, I am actually very, very keenly interested in. Um, and this is from, uh, Tom, uh, we'll just stick. Uh, he said, uh, how do you reconcile a healthy now next later planning model with the customer's laser focus on fixed delivery dates? You know, I remember, uh, in my day, the trip, our from 1997 to 2010, we floated the notion of, uh, a roadmap with no dates, uh, something maybe not as pithy as now, next later. And we got laughed out of the room. It was so alien, right. Not just internally, but externally. Uh, I would just love to hear, uh, how does one respond to, um, and make a defensible argument that, uh, this is really what we want anybody.

01:25:07

Okay. Okay. I can try it. Uh, so I think that, you know, I certainly know as flow improves, the business needed for dates goes down. Now that doesn't, that's not to say it doesn't go away. Uh, but I think what we've observed is that every day is it's a BCF future. It's a commitment on a value stream. That's significant constraints, the value stream. That's not necessarily connected to the outcome because it might be for one customer not you're not 50, um, and might prevent doing something great for 50 customers because you've committed this to one customer. Uh, so when one business partner or something of that sort. So I think the, you know, the approach that we have is just, just to understand how expensive those commitments are and that they're also stacking a form of debt that can bankrupt the value stream. You know, there's all this early stories from, from Oracle where the, just the number of bespoke customer commitments that they made around that database made it impossible to evolve anything for the whole market, because it was all being involved for these large strategic customers.

01:26:07

That was something that a couple of decades ago really, uh, really, uh, struck me as something I never in, anything I was ever involved in. So, so I think that the key thing is just to focus on flow, to understand how much each of those roadmap commitments. And by the way, I've messed this up repeatedly. I remember when I was like seven years ago, when someone actually did an analysis of all the customer commitments that we've made, and it was actually unachievable, it was not possible to deliver all of them within two X, the capacity that we had. So, so it's easy to fall into it. But I think that just understanding the economics of those commitments, um, is really important. And of course, this is where again, the OTRs are a great tool, because if you're focused on outcomes, you're doing that together with a business. Then you say, okay, if we make this roadmap commitment, guess what's going to happen to these other outcomes and guess what might happen to our overall revenue and, and exposing those economics because that's, what's so hard, is that how expensive a commitment is for teams to make, especially for 20 of them not to, and how that actually will impede driving the outcomes that everyone's after.

01:27:12

It takes away your decision, space, optionality and so forth. Right? Absolutely.

01:27:17

You can learn basically you've committed to anything you learned between now and that commitment you can act on because if the commitments so good,

01:27:25

So good. So actually, um, I, um, uh, I feel like I need to make sure that we hand this over to James to make sure that we can transition into the next, uh, element of this, which is the after party. So I just want to thank all the panelists, everyone who asked questions on and so forth. So, uh, I'm, I'm hoping this was, uh, sufficiently exothermic. Uh, we will assess whether, uh, we can do this again. So with that, I can turn it over to James, but only after I think, um, Julia Harrison, who joined given a very short notice, uh, Dr. Kirsten and John smart. Thank you so much. And I'm gonna turn it over to James.

01:27:59

Thanks, Jean. Yeah. Thanks guys. One amazing session. Really good to see the, uh, the speakers talking, ask each other questions, discussing the points. So the plan is we're going to head over to the gathered town. If anyone feels like coming along saying hi, chatting, finding out about his experiences. I don't mind I'll be there. Nick will be there. There'll be a few other people there. Come and join us. Let's see what happens. Okay. So thanks guys. Great session link should be in the chat by the way. I think that's going to be posted up now. So yeah.