True North and Technical Debt

Welcome everyone, and thanks for coming.

It's the last day of our conference, so yeah, thanks for not taking the last half day as a holiday.

Okay, so I'm here to talk to you about true north and technical debt.

I work at a business called Findmypast, and we run a bunch of websites that help people trace their family history.

This isn't really a talk about what we do, but how we do it, and it's a talk about problems.

So I want you to cast your mind back to when maybe you were a developer in ops, or maybe you still are, and that's great.

So I want you to imagine coming to work on a Monday morning, and you're feeling pretty good about it.

It's going to be a good Monday.

And you have your standup with your team.

You pick up an easy one-pointer, or what you think is an easy one-pointer, and you get started.

So you start having a little look at the code, and actually, it's a lot more difficult than you thought.

It's only changing your URL, but the business logic underlying it is really complicated.

It takes you all morning to do it, and you think to yourself, "This should've been easier," but it's normal that it's not.

You commit locally, and then you type in these keystrokes.

You type G-I-T space P-U-L-L, and this sense of dread comes over you.

You hit Enter, and your worst fears are realized: 47 merge errors.

So you just stop, and you go for lunch.

You're not doing this now.

You come back after lunch refreshed and make a nice steaming hot mug of coffee to sit with your steaming pile of merge errors.

You get your shovel, and you start digging.

So, the merge errors take most of the afternoon to deal with, and you think to yourself, "This should be easier," but this is normal.

You eventually get to the end of the day.

You commit the code.

It's 5:00.

You think, "I'm not going to push this now.

It's too risky." So you come in the next day, you push your code, and everything's going okay.

And it's going to acceptance, and then the test fail.

And you read the test, and you're like, "Why has this test failed?

I'll just run it again." Run it again, another test fails.

Not that last one, but a different one.

Maybe run it again, and it's a different test.

And you're like, "Ah, third time lucky.

Fourth time lucky.

I got through." So yeah.

But this is normal.

And then finally, you get your work out to production.

You've spent a day and a half of what you thought was going to take 10 minutes.

A fleeting thought comes over you, "This should've been easier," but this is normal.

And that's where we were, and it got me thinking about why don't we get faster?

Why do we get slower?

I've been in many tech teams where the delivery rate looks like this over time.

As you add complexity and the systems get harder to work with, delivery gets harder.

And four years ago at Findmypast, I started looking at these kind of problems.

So the first one was we don't fix our problems.

I'll give you a couple of examples from my history.

When I started out as a games QA, we would come in at 9:00 every morning, and it would take one hour to burn the disks for yesterday's build.

I don't think we ever saw it as a problem of wasting 10 hours every morning.

And then I worked in another company where it was the norm to spend two days a week merging code.

This was normal, and we didn't see it as a problem.

And we never fixed those problems, and those businesses don't exist now.

And I've seen this pattern as well, where delivery rate gets worse, and then you work hard to kind of stop, and then you fix a bunch of stuff, but then it gets worse again.

And it's this constant goal conflict of right here, right now, the product that you want to deliver this week versus your long-term goals of working on the system of work.

And we're addicted to complexity.

This is something else that I've seen is that we like it when we develop a complex solution.

It makes us feel clever.

And I recently read this book by a guy called Dan Ward.

It's called "The Simplicity Cycle", and he says whenever you build anything, you should look at the goodness of what you're building and also the complexity.

In order to add goodness, you need to add complexity.

But there is an inflection point where adding more complexity makes it less good.

I bet you can all think of a product that was simple and is now complex, and you think it's now worse.

Microsoft Word.

And what you need to do is simplify.

And we do what's best for us.

So this is about making a decision within a set of bounds, a rational decision.

I recently had to sit down and figure out how many content management systems we have at Findmypast.

I started putting the feelers out.

I found one, then two, and then three and four, and five and six, and seven and eight.

And do you want to know the worst thing was?

I added two of them.

And this is not better for the whole, but it was better for the right now.

We all made those decisions in a rational way.

We had a delivery that we had to hit.

We had people that needed this functionality, so we just added and added and added.

And this is what we did.

We made it worse by making it more complex.

And we're obsessed with workarounds.

So I mentioned earlier in that little story about the tests, about rerunning them.

That is a workaround.

And you become normalized to that deviance, and it just becomes normal, and you don't even see them as workarounds anymore.

And we disagree about everything.

Four years ago, I remember having this really spirited discussion with somebody about whether or not it was worthwhile automating copying code onto servers.

They felt that it only takes five minutes.

I can just drag and drop the files.

It's fine.

I've been doing this for years.

But that isn't a good use of human ingenuity.

It's just toil.

It's just a waste of time.

It's not worth our while.

We should automate that kind of stuff.

But why were we even disagreeing about it in the first place?

And all of these things lead to a feeling of powerlessness.

When you face all these problems every day, when this weight of the system, you just don't feel that you can get stuff done.

If you're leaving at the end of each day not feeling like you've delivered anything worthwhile, like you have a purpose for being there, we need to do something about that.

And we were utterly, utterly lost, I feel.

So what did we do?

Finding our true north.

And what is a true north?

So a true north, I mentioned, is about the how.

So not the why of your business, not why you exist, not the purpose, and not what you're building, but the how.

How are you going to go about and do this work?

And this is a concept that came from Toyota.

And what it is, is it's this long-range target of how, how you're going to do something.

So I'm going to show you what Toyota's true north is.

Zero defects, 100% value added, one piece flow, and security for people.

You'll notice there's nothing about cars here or sewing machines.

This is about how they're going to do the work.

And you'll also notice that these are all impossible to attain.

We live in a non-deterministic world.

There are going to be accidents, but that doesn't mean that we shouldn't aim for complete security for people.

And that's what this is about.

It's about giving us direction.

So we have the CALMS model from DevOps, and this is the culture, automation, lean, measurement, and sharing.

So this is from John Willis and Damon Edwards.

And then Jez Humble added the lean.

This is from Lean teachings.

And I first read about this in Toyota Kata.

I would recommend that everyone read that book.

I think it's fantastic.

And there's a follow-up manual about how to build a coaching culture of problem-solving.

And that's where I found out about this.

And I said, "We need one of these at Findmypast." So our true north is our long-range unachievable target of how we build brilliant software.

And this was going to help with some of these problems that I mentioned before.

So we started off with a tech and product summit with nine options and nine pitches.

So I surveyed a bunch of different engineers and product people from the business and kind of got together nine options.

I presented to them what I thought a true north was and why it was important.

And then they self-selected to one of these options and then pitched their principle back to the team, and then we were going to vote on it.

So we got all of tech and product together, about 60 people, and we ran this as kind of a two-day internal conference.

And then the output of that was going to be our true north.

So these were the options.

Automate everything, continuous and instant deployment, a frictionless experience for all.

Zero defects, 100% uptime, monitor, alert, and visualize everything.

100% business value, something else entirely, or always improving, always experimenting.

And this is what our true north turned out to be.

This is what everyone voted for, was monitor, alert, and visualize everything, a frictionless experience for all, and always improving, always experiment.

Here's where we made our first big mistake.

We didn't support the change.

We got everyone really excited about this, but then we didn't look about how we were going to build that into our day-to-day work and into our processes.

We didn't think about that at all.

We created this kind of wave of enthusiasm but then lost all momentum because we didn't support the change.

And we needed time.

We'd created even more goal conflict between these principles, these long-term principles that you want to work towards, and the short-term of delivery.

So this is about convincing the business to give us some time.

And when I have to go and pitch for something like this, which is a big change to process of how we work, I form this kind of mental model or a theory of mind about how I expect them to think.

And for some reason, this is how I think execs think.

But no, they're always asking about the value of things.

So I set about creating an argument to see whether they would agree with me on certain principles.

So I'm including here two premises that I posed to them.

So the first is that this is a day, and that a day is filled with four types of work.

So the first one is innovation.

This is what drives the value for the business.

The second one is firefighting.

This is where we disrupt people's work.

For us, this is mainly incidents, but it could be something like, somebody comes and taps me on the shoulder and says, "Somebody's pulled out of an interview.

Would you come and do it?" And I have to drop everything and go and do that.

That would be classed as firefighting as well.

And these four types of work are similar to what you read about in "The Phoenix Project." We've got toil.

Toil is the tax you pay to deliver some innovation.

So in that story I told you earlier, there was maybe 10 minutes of innovation and then a day and a half of toil.

And the final one is improvement.

So where is the waste in this system?

It is toil and firefighting.

But how much of each type do you think that you spend?

So this is what we might like to be true.

With something like 75% innovation.

I don't know how many of you feel like you get to spend this much time on innovation.

Yeah.

I thought so.

So here's the reality.

It's maybe a lot more toil.

And then I pose the question: how do we make more space for innovation?

The only answer is we have to reduce the time we're spending firefighting and on toil.

And how do we do that?

We improve the system.

So it's about improvement time focused on those two things.

So constant and deliberate improvement to reduce toil and firefighting to make room for more innovation.

And then you stop and you say, "Can you agree with this premise?

Do you believe this to be true?" They agreed with me, so we can move on.

And the second thing is about intuition.

I'd like to do a quick survey.

Who in the audience has a mortgage?

Okay, great.

Okay.

Who in the audience knows how long they've got left on the mortgage?

Okay, great.

If you were to overpay that mortgage at £250 a month, do you know how long you'd have left on your mortgage?

This guy.

So, one.

So this is about intuition about compound interest or exponentials.

We're really bad at it.

So with no improvement, I want you to imagine a team delivering an abstract of 10 value per week.

This is about as mathsy as it gets.

So this is linear, this is intuitive.

At the end of the third week, you've got 30 value.

At 1% improvement, I'm trading two value for 1% week-on-week improvement, or 20% of time.

Okay?

So this is how much value you have after the third week.

It's not so intuitive.

Okay?

That's compound interest.

But what is it on a yearly basis?

So that's the short term.

This is the long term.

This is easy.

520, 1040, 1560, 2080.

This is linear, again, intuitive, nice and easy.

Can any of you do the 1% week on week in your head?

Yeah .

So 542, 1497, 2978, 5537.

But what are the actual compound gains here?

4% in the first year, 44%, 91%, and 166%.

But what is 1%?

That was something that I was asking myself.

On a really good day, an engineer might get five hours of solid work done.

Five hours is 300 minutes.

1% is three minutes.

Over a week, that's 15 minutes.

Do you think you could spend one day and find 15 minutes of toil to get rid of?

If you did that every week, continuously, at the end of four years, you'd be 166% better off.

Do you not then have to do that once you know that, if you agree with the premises that I'm putting forward?

So I got the green light and we went ahead, and this is about how we then prove the point.

So measure it.

We did a little experiment with one team in one quarter, and we put up a sheet like this, and every morning at stand-up, they said how they'd spent their time yesterday.

So they got a couple of check marks, and we ended up with something like this.

This turned into a graph over time.

And can prove the point that with improvement, we could switch the balance from toil to innovation.

You'll notice that in the first few weeks, they didn't feel like they were spending time on improvement, and that's because they were maybe only doing about 5 or 10%, so they weren't putting the check marks up.

So we proved the point, and then we got the go-ahead to roll it out across all of engineering, and this was to spend 20% of their time on improvement.

So what we did was we got the teams on board, and we reiterated the types of work, the mental models that they were going to use, the true north principles about how work was going to be done.

We started measuring.

So the teams all started measuring, and then they got their graphs as well and started to plan improvements from what they were identifying for the causes of toil and firefighting.

But we did create some time conflict here, so this didn't work for all the teams.

Some teams did it on a backlog basis, where they put 20% in their backlog.

But it tended to be that they always got deprioritized again, and then the short-term delivery goals was what people went after.

So we ended up with a true north day.

We went to a bimodal way of working.

Quite a few teams do it on a Monday or a Friday.

They do their true north day, and they have their specific toil and firefighting things that they're going after with that time, and we found that to be more successful than the other way.

And now I want to talk to you about taking it a little further.

So this is about experiments that we're running right now.

So the first thing I want to ask you in regard to this is, how good is your working memory?

If I was to ask you exactly what I said on slide three, could you tell me?

Right.

Working memory is only two minutes long, so you can't remember the exact thing that I said, but you'll maybe remember the feeling.

You'll maybe remember the idea, but you won't remember exactly what happened.

And this is about the perishability of information after the fact.

That's why you're supposed to do postmortems as soon as possible after the incident is because the information will perish.

We will forget what happened.

So the first thing is about surfacing problems.

This is how we surface problems now.

We have three colors of card on every engineer's desk.

We have yellow ones or orange, and pink ones, and blue ones.

The yellow ones are for toil, pink ones are for firefighting, and the blue ones are for something a little different.

They're for process problems or autonomy problems, like I need to go and ask permission because I don't feel confident to make this decision.

And what we end up with are these, these shared problem walls.

So in Dundee, we're based over three floors.

At the door you enter the floor, it's this tower building.

It's this glass panel.

We put up the problems on the wall, so you have to walk past them as you're going in and out of the door.

And this has some interesting side effects that people read about them because they're up there, they're in their face, it's visible.

People start having conversations about these problems as well.

So we have all these shared problem walls about the people are raising issues that they're having with work.

Okay?

And then what we do is we swarm them.

So every Friday at 12 o'clock, we get the entire engineering team, that's 35 engineers in Dundee, and we bring them all up to floor nine, and then we talk about the problems on the wall.

Some of them get solved right there and then.

Somebody just knows the answer.

Some of them are harder to deal with.

They're big cross-team efforts.

And what we do with those is we see people are really interested in solving them, but what we do is we start a swarm in Slack, and we invite everyone in, and then we let people use their true north day next week to tackle that particular problem.

And you see some really interesting cross-pollination across the teams, and that's worked really well for us so far.

We're about five weeks into that, and we've already solved some really big cross-team problems.

We had 104 toggles in our UI code base.

That's now down to 50.

The complexity there was rising, and the engineers identified that and dealt with it themselves.

But that was a proper cross-team collaborative effort.

So it seems to be working.

And then you have the various people involved.

So, the last bit that we've been doing is coaching and learning.

So our teams all run a weekly or a fortnightly retrospective.

One of the questions that we've started asking in that retrospective is, look at all of the work that you've done in the past week or two weeks.

What would need to change, hypothetically, to deliver that work in half the time?

Everything's on the table.

And that leads to some really interesting ideas about what changes we could make in our system, and lots more of these cards go up on the wall.

It tends to be there's more process or autonomy things come out of those.

And then finally, we have monthly coaching with the team leads about embedding these practices within the wider team.

So we have the weekly cycle with the team and then the cross-team cycle on a monthly basis.

And it's about deliberately practicing problem-solving and making it part of everything that you do.

So what have we learned?

We've learned that I can't remember what I said two minutes ago.

We've learned that we're all really bad at maths when it comes to compound interest, and that we all have a lot of problems to solve.

And I want to leave you with one final question.

What game are you playing?

Are you playing a game that ends in three months?

Are you trying to win a three-month game?

Or do you think your business is going to be here in four years' time?

Or is it going to be here in 100 years' time?

In terms of Findmypast, people are always going to want to know about their history.

We are playing in an infinite game, not a three-month game.

So you have to adjust what rules you are playing to.

And that's where I think a true north and deliberate problem-solving can really help.

Thank you very much.

I think we've got time, four minutes for questions if anyone has any.

Yeah.

Actually, so how did you go about measuring the amount of the work percentages that you presented?

How did we go about measuring it?

Yeah.

So with the check marks, at the end of every week, we put in the percentage of work that people were spending, and then we tracked it.

And we saw that generally, the teams over time could identify where they were spending time on toil, tackle that, and once they tackle that, they were able to deliver more value or spend more time, at least qualitatively, on innovation.

For example, developers would self-identify, "On this story, I spent like X amount of time doing toil, and next month, we prioritize it and spend more time on it." Yeah.

But if I'm being honest, that didn't actually work that well.

It worked to prove a point, but the problem walls work a lot better because you've got the cards there, and as you experience the problem, you just write it down, and then you put it up.

And the whole thing about swarming it and getting people talking about it is much more useful than measuring it.

We could measure it from the number of cards on the wall probably, but we've not been bothering at this point.

We've just been attacking what we deem as the highest priority problems.

So does that answer your question?

Okay, great.

Okay, I think we're done.

Thanks, everyone.

Thank you.