Las Vegas 2022
No slides available

Modularizing the Messy Matrix

Presentation by Mik Kersten


Mik Kersten

CTO, Planview Inc.



All right, thank you. The next speaker is someone who should be familiar to all of us in this community. Dr. McKeson wrote the awesome book Project to product four years ago, which is being used by so many organizations, uh, to change how they think about their software efforts. So Mick taught me so much about architecture, which is so important to enabling fast flow, which I think this community understands better than almost any. In fact, this set the stage for the work that I've been doing with my mentor, Dr. Steven Spear, on how important architecture and the structure of the organization is to get the outcomes that we want. So we know that we can create a structure that fully unleashes the full creativity and problem solving capability of the organization, or we can create one that suppresses or extinguishes it entirely. But, uh, one other aspect, uh, is important, which is signals are, can be amplified or signals can be suppressed.


And these are the signals that leaders need to know whether, uh, the system is actually working regardless of how good the architecture is. So I'm so excited that Dr. Mick Kiersten will be presenting a real life case study of a reorganization that he did at his company. He'll state the benefits that he expected and what actually happened and what he did about it. So I think this is such an important talk, be he, because he models so explicitly how we need to think about organizational design, and it's something I've rarely seen, but I think it is so important to, uh, get the outcomes that we want. So here's Mick




Hello everyone. I'm thrilled to be here at my favorite conference with all of you in person and building these new memories and relationships that I think will keep us going through the next year of Zoom calls. And, uh, it was, it was, as Jean said, four years ago, that project to product was launched here. Uh, since that time, I've had a chance to learn a lot around what's worked, what's been easy, what's been challenging about helping organizations shift and really elevate these principles of DevOps and Agile to the business as a whole. So over that time, I also, my own careers has, has changed as well. I started as a developer where I actually found it very easy to look at how we were able to transform the, our software architecture and open source projects, transform the team structure, and really change around our value streams as our projects evolved.


But as I stopped doing actual work and, and just started helping other people do work and take burden out of, out of their days, uh, as I shifted from being a, a developer to being a C E O for, for 15 years, most recently, uh, with the acquisition of Task Up by Planview of, I've now become a Chief Technology Officer for planview. So my scope in terms of engineering has, has grown some more. I've realized I've needed the same kinds of tools I was using for software to better understand what was working about organizational structure. And it was really interesting actually, the, the past few talks, we've heard a lot about abstractions, about modularity and about organizational and, and functional design. So I've been applying these in this management system that, that I've applied a task up and now at planview to looking at how we can better measure and understand organizational structures to see are they making work easier for staff or, or are we actually adding burden and toil into the systems?


And so I'll talk to you about that today. But really from the perspective of how we measure the flow of the, in through the organizations that we create, how we measure these signals for the structures that we've set up. Uh, and without giving so much advice on what structures you wanna end up on, but really how you take a disciplined and data-driven approach to better understanding flow. So the really good thing I think over the past four years is that I have had a chance to witness how this shift of project to product that so many in the community have been working on has helped catalyze change. Uh, we've got a whole new kind of alignment in terms of business outcomes that was just wasn't there before. When we were kinda throwing work onto agile teams. A lot of organizations have actually changed how they budget and plan and take a whole new approach to, to strategic, uh, portfolio management with their software of funding value streams and having fixed capacity.


And then there's been this really interesting, I think, evolution of organizational design, because a lot of organizations have realized agile teams are not enough, and agile teams are kind of easy to deploy, but business agility is, is a little bit harder. And so all of this, I've realized centers around this notion of, of value streams and actually defining and refining how we manage and measure our value streams and the value streams we know need to be aligned to the customer to delivering business value. Uh, they need to be autonomous. And as you'll see in the talk, as when they become less autonomous, the flow of value slows down. They need to be cross-functional because we can't squeeze all that cross functioning into a single team, as you probably know this from Christophe's, uh, previous stock on Google, SS r e. And of course, they need to be customer centric.


They need to know what their customer are their their customer is. Now the challenging thing is, and for a lot of my work has been supporting organizations that work at very large scale scales, is that these value streams need to span functions. So span disciplines like engineering, architecture, design, business analysis, ss, r e, uh, portfolio management, they need to span hierarchies because a lot of our organizations have created these functional hierarchies. They need to span geographies, and oftentimes they need to span sourcing partners as well when you've got that kind of global complexity. And so this problem, I, I actually heard Charlie Betts, who's in the audience here today, uh, coined the messy matrix. And so we're always looking at how can we structure the value streams given the organizational design that we have to today, uh, in order to make them effective, given that dude, we're gonna cross organizational boundaries and silos and functions.


And I think the challenge is for a lot of organizations, this has just not been done effectively. So the result of this is we've got well-defined strategies and all the best intentions of creating the best customer experiences, uh, delivering value to the business. We've generally got well-formed teams because agile teams are, are effectively a solved problem. But this middle layer of value streams, we need flow. We need to track business outcomes where we actually need to make sure that we we're tracking to the happiness and, uh, of our staff working on of those value streams tends not to be defined, tends not to be properly formalized, and is actually suppressing all sorts of signals from being fed back up to the strategy. So those signals around how much we should be, uh, investing in technical debt, when we should be replatforming, what's putting toil into the systems, which organizational structures are working, which ones aren't, tend not to percolate up when value streams are not effectively defined.


And when we don't have effective ownership of them. Uh, and then that means that we're lacking the alignment of actually making sure that the strategy is connected to delivery, that the teams are working on the right things, that we've actually properly staffed our platform teams, because typically they, they tend to be understaffed, right? Uh, a lot of the resources and teams get put on, on the business facing products. Not enough gets put on the platform teams to actually help accelerate the delivery of those products. So this disconnect I've seen prevalent across the, I think I think the majority of enterprise organizations. I'll show you just some quick data of some of the effects that we've seen on this by collecting, uh, this is data from over 3000 value streams across 40 large enterprise organizations, uh, collected through the plan view task of this tool. So analysis, some data mining that we've done on this.


And what we're seeing, and this is this, this is, this will give you a sense of how problematic this is, is that across those value streams, 8% of what was planned by agile teams was delivered in the, in the timeframe was planned for. So there's this complete disconnect in terms of how we're planning, uh, and the demand that's there to innovate for customers and the actual capacity of teams working. The next one is even more bothersome to me. Uh, 20% of features are canceled after code has been written. So you know how demotivating this is. If you've ever written code, when the business changes its mind or something's changed in the strategic plan, because again, these things are, are disconnected the way that planning's being done and the way that teams are working. Uh, 35% of products have zero capacity for new work for the next 12 months because they're undergoing all this change.


And again, we've got these disconnects and then the disconnects in terms of the signals. 85% of products under invest in security and debt, because of course that those priorities not making it up when we don't have this properly formalized, structured, uh, later of the value stream. And so I do think there's a better way. I think that the work that Gene Kim and Steve Spear are doing right now in terms of better understanding how dynamic learning organizations work and how to formalize this structure and dynamics points the path forward. And for me, the, the real aha moment came when it actually was, when I first met Gene in 2016 at a conference, I, I thought I have to meet him. I, I was, uh, just amazed by his work and show him this visualization I had just made of what I thought good look like.


And this was, uh, visualization using the Gores tool. Jean, you, you might, uh, have a little flashback here. Um, the, uh, showing the commit history, the source code history for the Eclipse Myan project I was working on over several years. And what's happening here is you're seeing all the commits and pull requests. And as this is happening, the modular structure of this code base is forming. It's actually a large code base. Uh, it was evolving quickly. It had a lot of, a lot of committers. And what we're seeing is that encapsulation, what we saw, what we heard Christina talk about, where these modules are forming, they're encapsulating, they're hiding the right kind of information. And you'll actually see this, uh, these orange rays darting out. Those orange rays are me. 'cause I'm constantly refactoring the code base, realizing the encapsulation wrong. I'm seeing signals that we're not hiding the right information.


These APIs are not quite right. So I'm constantly doing all these refactoring to make the work of other contributors and other teams using this code base easier. And that, of course, relies on me getting the right kinds of signals of, is the encapsulation right or is it wrong? Where do I change, where do we change it? Where do we make it better for people to contribute? So as this is going on, you actually get a, get a sense of the encapsulation improving. This, this open source project for its time was actually very successful. And it gave me a sense of what good looks like in terms of making sure it's easy for people to contribute value and to maximize flow. So in terms of good modularity, one of the main things that we rely on is high cohesion. So every single module, uh, encapsulates common functionality and that that functionality can be really used as building blocks for our software and give us that, that nice feeling of, uh, of Lego as we're building software and being able to recompose it.


Um, and of course, the whole goal of software, software modularity to manage the immense complexity that we're dealing with. So high cohesion is, is critical in doing that. And then low coupling, low coupling is the other key, uh, key principles that we need because we need to decrease dependencies between modules in order to be able to evolve them independently, uh, in order to hide information so that, so that we're, we've got the right layers of abstraction. And then to avoid these cases where we get into these spaghetti code messes. And what I realized is the same kinds of things apply to value streams and organizations where we need to encapsulate expertise and ownership, and we need to decrease dependencies between value streams. The more we decrease them, the more autonomous they are, the faster they can go in, the faster they can move. And so really the role of leadership, and this is where, where Jean and Steve have these concepts of configuring the organization and then running the organization.


So when we're configuring the organization, we de we're designing it, and then we're assessing it and we're assessing it. Did, did the configuration work or do we actually need to change the encapsulation? The principles, uh, for that are of structure. So the way that we structure our hierarchies, the way we structure dotted lines and matrices in the organization, and then the dynamics, the signals, how they flow as people work, was it easier for them to do work or they have to go to 10, 10 people to get permission to do work? And we've got these new toolkits that have come actually outta this community. So team topologies, other approaches to organization design give us a common vocabulary, uh, as well. And the flow framework, and this is really my, my main goal with project to product, uh, makes it easy to measure these kinds of changes as as as we're making the shift.


So now I'll quickly take you through that, a task that we, your story. So four years ago, uh, as we, we were obviously evolving our shift, growing quickly, our shift from project to product, and our hypothesis was that we should actually merge the product and engineering organizations into one organization, all hard lined, everyone point to one part of the organization. And that this will accelerate flow. Uh, the expected outcomes were we, we'd have more customer focus, less us versus them platforms. All platforms would be elevated to as first class products, just like our customer facing products. And of course, we'd use the flow framework to see did this work? Did it accelerate flow or did it not? Uh, now in terms of the assessment, I could start getting a feeling last summer that it wasn't working exactly as expected. What was happening in terms of the flow frameworks, business metrics, the value was there.


So the vis products AR had grown over 200% year on year. The, the revenue, uh, the cost, we, the, the team size had doubled the hosting cost growth across it was about two x, which is okay, but the challenge was that the flow velocity was trending to, to basically be flat even when we compensated for the onboarding time. So velocity, the flow was not accelerating. So something was off. And then the happiness was showing a steady decline, the happiness of the staff working on the product, all of those various teams. So the diagnosis as we dug in deeper was the cohesion had been dramatically reduced and the teams were constantly reporting all this cognitive overload of having too much work put on them, too many complexity domains put on them. Um, and the coupling had just gone through the roof. And I just will never forget this moment where it was announced the whole, our whole product and engineering organization that we have to 10 x the amount of collaboration between value streams, because that's actually an anti-pattern if you've got that much dependencies between those value streams.


So it was time for refactoring. And so we refactor the organization earlier this year in, in February, uh, where we again, split out product and engineering. So engineering now had, its had engineering leaders. Uh, the organization was still very much matrixed in the sense that value streams are everyone's first team, uh, SREs are more embedded. And again, the point is not exactly where you end up in this matrix. The point is, is the structure meeting your business needs? Is it making work easier for people? In our case, it was sort, it's a similar thing to what Amazon noticed with two pizza teams and single threaded ownership. We kind of fell for the theoretical ideal of, of single threaded ownership of everything being in one, in one organization. What worked better for us is to actually split these things apart. And then we realized we were act, we were making much better progress, uh, on our goals of re-platforming.


So really the whole goal here is to put this in place. You, you, you need to formalize these value streams. You need to measure, are we moving faster? Are we moving slower? Um, you need to make sure that those signals are getting to you. My challenge was that not enough signals were getting to me other than through these metrics because a lot of signals were being, so engineering signals were being suppressed at the time. And again, measure those flows and outcomes and connect delivery, uh, strategy to delivery. So quick, very quickly to summarize, define your value streams. Again, we've got the toolkits there. Team topology, single friended ownership. You'll hear, hear Dean Leffingwell talking here about the scaled Agile framework. Ss r e, those various models. And then make sure you're constantly assessing what you've, what you've defined via flow metrics, measuring business outcomes and as, as well as the happiness of the staff, which is a really important leading indicator. And then the key thing is continually refactor that. So you need low coupling and high cohesion for product value streams, your organizational design and your software architecture. And really think of those as one and apply those principles modularity so you can learn more. Uh, check out our, which makes this easier. Check out the podcast where I talk to a lot of my mentors about this. And the help I'm looking for is if you could document some of your own experiences measuring flow and organizational design, uh, with a flow frame of community. Thank you.