Green, Green, Green, Red
Dame Emily Lawson and Mallory Durran tell how the Prime Minister's Delivery Unit tried to see which major government programmes were about to go wrong. Monthly reports stayed green, green, green, then red. They trained a model on the language of programme directors, ranked where leaders should look each month, and built a feedback loop that rewarded honest reporting, and they share what they learned when the approach was dismantled.
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
All right. Our next two speakers worked together at the very center of UK government. Dame Emily Lawson is Chief Operating Officer of Newton, who helped make this event happen. Before Newton, among her many other achievements, Emily ran the UK's COVID vaccine program and then led the delivery unit at Number 10 Downing Street.
So when I met Emily this August, I told her about a friend of mine, Josh Cohen, who worked inside of America's Warp Speed, whose mission was to develop and distribute COVID vaccines. And to this day, he still holds a grudge against the cheesemakers in Wisconsin because they had a lock on the US nation's dry ice supply. And when I told Emily this, she said, "I remember that. I was on the phone with those guys many, many times."
So Emily told me that she's not interested in technology for its own sake. Instead, she's interested in getting things done that make the world better. And she told me about one of her colleagues, whose work she depended upon, who she called an all-around brilliant human.
And that's Mallory Durran, Chief Technology Officer of Nesta, UK's Innovation Foundation. She's a data engineer by training. She came up through hacker camps in Canada and spent seven years in UK government serving four prime ministers. And the motto of her team was making sure the most important decisions have the best evidence.
So here to tell us about their important work, please welcome Emily and Mallory.
Emily Lawson
Good afternoon. And I was thinking about, in this sort of gentle post-lunch, had a couple of talks, sort of moving into passive listening phase, what is the reason for you to pay attention for the next 20 minutes to what Mallory and I are going to talk about?
And I'm hoping this is compelling, but we want to pick up on a theme that was started by Jacqueline this morning, and I think has been implicit or explicit through all of the talks we've heard so far, which is: what is the role of leadership in making the most of this extraordinary technology that we've got at our fingertips?
And I wanted to start from something that a CEO I used to work with said to me about 20 years ago. And he said, "Emily, the thing is, the more power you get, the more you have to ask yourself the question, 'What are they not telling me?'"
And I think—hope by the end of this, I'd like some responses on this—but I think, for me, that question's morphed into less, "What are they not telling me?" but, "What is the question that I'm not asking?"
So what we wanted to—I think this might be going—but what we wanted to spend time on in our cryptically entitled conversation around "Green, Green, Green, Red" is what happens when you do try and ask the right question.
And if you think about the systems that Gene just mentioned in the introduction that I came into this particular story having run, so the COVID rollout program and then leading delivery across government, both of those are enormously complex systems where, as a single leader, I had absolutely no hope of looking at the system in its entirety and understanding what was going on.
So if I've got no hope of looking at that entire system, and my experience is that as I see reports come up through the organization, I see them not necessarily bringing the truth with them, what I wanted to do in the system that I found myself in was figure out: how could I actually get the truth out of those reports?
And even more relevant now, four years later, where we have increasingly sophisticated AI producing, if we're not careful, increasingly misleading reports because we aren't asking the right question.
And what we want to tell you about is a story where we think we managed to put together the leadership, the asking the question, the team, and the machine in a way that became a positively reinforcing loop of actually asking the question and, in doing so, generating more powerful answers.
So we couldn't really resist taking you back in time even further. And we didn't know we were coming after actually having seen the machine. But our starting point was this idea, as it was at Bletchley, was: what was the crib? And once we had the crib, what did we do with it? Mallory.
Mallory Durran
Yeah. So you've just heard an amazing talk from Bedfordshire about what they're doing in policing. But pre-large language models, and indeed in many of the 43 forces across the country, you wouldn't have had a leader like the one you have just heard from.
So in 2018, 2019, this is a very crude diagram of what was a network model that we built after scraping adult services websites. I'll let you fill in the gaps. Using machine learning algorithms to identify networked phone numbers, contact information, and language that were likely to be associated with exploitation from organized, or at least moderately organized, criminal groups.
We built this network model. We had loads and loads of historic data from forces, from the National Crime Agency, that had been collected. We could validate the accuracy of that model and say, actually, casting forward with 99% accuracy, we think that every network we have identified is associated with exploitation occurring live now in the country.
Despite the fact that this is a horrific crime type and we had massive confidence in what we were identifying, that translated into zero dollars or zero pounds of operational spend for a wide variety of reasons.
Some of those were operational constraints, but a lot of it was, "Well, you validated this using statistical methods. You know it worked in the past, but crime types change all of the time. We don't have the kind of confidence in what you've built to be able to move with that and for that."
And there wasn't the leadership or the relationship between model builders and model users to be able to actually translate that into operational success.
Emily Lawson
So in the 2022 world that we're going to tell you about, I was the Director General responsible for the Prime Minister's Delivery Unit, but also all of delivery across government. So it didn't all report into me, but it was my job to pull all of that together, including all of the support from the data science team and from the i.AI incubator.
And what I found was, in the Government Strategic Management Office, or GSMO, there were 235 cross-government projects that were being reported into that team in the Cabinet Office every month. Not every team reported every month, but every team was supposed to report every month. And Dominic Lake, who ran that team, and his team were sucking that data in through a cross-Whitehall system and producing these reports.
And a lot of the time, those reports were green. And what I was interested in is: what were we doing with that information?
First problem was that the reports were being produced inconsistently, so not every team reported every month, and which teams reported and which teams didn't was in itself inconsistent.
And the Cabinet Office team were a very well-managed team, and they built a great system, and they were constantly improving it, and they reported that data. And that data was incorporated into reports within the Cabinet Office, but also a report that was framed by the Number 10 policy team that went to the Prime Minister also every month that reported on the Prime Minister's priorities.
And those reports were boxed for the Prime Minister, and I think there are enough Whitehall people here to know what being boxed means. So the box went to the Prime Minister, and at some point, the Prime Minister either read that report or didn't read the report.
And sometimes I saw some of the reports come back with comments written on them, and the comments were often, maybe—I think often's reasonable. I'm going to check what Mallory thinks. She's making a face now. But the feedback that came back was usually on one of maybe 30 points that were made in the report, and often about policy, because, again, those of you who've worked in Whitehall, policy, it's an attractive lever, right? It's something you can point to. You can say, "Look, we've done something about this."
They were very rarely or almost never about delivery. And so we had this system of 235 reports being produced inconsistently, being reported, being boxed, and then intervention was limited.
So the green, green, green, red was the fact that what I observed with my reasonable but not in any way machine-like pattern recognition is it was almost impossible, reading those reports, to predict when that report was going to go red.
We also had the major projects being reported across Whitehall, and it was this chart that was the trigger for the conversation that Mallory and I then had that leads to this story.
So I don't know if you can see it at the back, but this is a direct screenshot from the IPA's 2021 report, and it's a Sankey diagram. I love Sankey diagrams. I think they're a brilliant way of showing data, but this one really upset me because the IPA was using it to say that, "Look at all this brilliant intervention, and look how much better all the projects are at the end of the year."
But I don't know about you, but that was not particularly what that diagram said to me at the time. And if you look at the actual correlation between the rating in 2020 and the rating in 2021, if you just look at the reds, what you can see is that having had a red rating this year actually told you very little about next year. So those that were rated red one year, about half stayed red, and about half had changed their rating to a whole range of other things, but not green.
So we were doing all of this gathering of information in the interests of delivering better for UK citizens. And think, the projects that are covered by this was everything from how better to prosecute and ensure better support for rape victims in rape trials, to the quality of teachers, to GP access, to the success of levelling-up policies in the north of England.
So this is a vast array of different kinds of things that are being measured, all of which had the potential to be hugely valuable for UK citizens. And so what I wanted to look for was, I felt there must be something in this data that was telling us the answer, but we just couldn't see it.
And that was when a conversation with Mallory completely changed the outlook and the possibilities that I thought were inherent in the problem. Mallory?
Mallory Durran
Meanwhile, I had built tens, maybe even hundreds of these kinds of surveillance systems, looking at unstructured data and trying to draw patterns out, but very rarely had they cut through. They got stuck in that gap between excellent model generating insight, and no one's picked it up and run with it.
So we worked with Emily in a very different way, which was that she had a really clear vision for how she needed that output or information to look, and what we needed to be able to identify in order for the information to be useful in the decision-making and application of support, triage, intervention to come out of it.
So we approached the training of a new set of models. Remember, this was 2022. We didn't have access yet to large language models, but I think all of the principles that we followed stand true today with how leaders can help technical teams develop and land systems that actually work.
So we were looking at a huge amount of historic data. So when we talked about the 235 programs, the monthly reporting, there were also more than 50 years' worth of major infrastructure projects and programs, similar reporting data available to us.
So we had what academics would call a pretty dreamy natural language processing set because we also knew what happened in those programs. So we could look at what was being said in hundreds and thousands of lines of text and translate that into: what did that mean for the program? What did we see? What happened?
So we didn't have large language models, so we were using things like transformer neural nets to look at language patterns, mapping that against structured data like RAG ratings and spend or overrun or missed milestones, and bringing those things together to spot patterns so that we could start drawing out, as Emily said, what the government language version of "Wetter" would be in our crib sheet.
What does it mean for a program director to be nervous or to be experiencing a change that wouldn't be necessarily drawn out, that is hidden 17 lines deep into their 42-page monthly report?
So we used a cross-section of expert hypotheses from folks like Emily, who had delivered programs in particular contexts, to say, "What did this look like?" or, "What did it mean to you?" And ended up coming up with what was, I think, in the beginning, a 20-metric rules-based algorithm that would rank-order score which of the programs we thought Emily and the Prime Minister and Cabinet should actually be drilling into that month on the basis of the language being used by program directors.
Now, the last part in this chain says, then we checked everything manually. With 20 criteria across 235 programs, some of those criteria being continuous rather than categorical, there was something to the order of 4.5 billion possible combinations. We obviously didn't test all of those, so we had to create a structured way to really challenge what we were finding with actual program directors.
So that feedback loop, I think, was one of the most important parts of the chain. We couldn't design a perfect test for every single metric or every single program, but we could do our work in the open. We could say, "Here's the rank order we came up with," approach program teams and say, "Is this an accurate reflection? Here are the 10 lines that flagged you within these 17 categories. What does that mean, and how do you feel about it?"
And what we realized over time is when those directors thought we were actually going to use their information, they improved the quality of their inputs. Go to the next.
So we started with the right question. I had built lots of these systems before. They never really had uptake, didn't know how to make them get used. So Emily's steer and commission were sort of the leaping-off point.
Then we put things into production. We built the system. We made the information available. We then used it in public. Now, in public in government can mean a lot of things, but we opened up both our code repository into an internal-to-government license, which meant any team in any department, whether programmatic or managerial or working for a minister, could see what we had done.
We made the algorithms transparent, and then we made the information we were producing transparent so that the program teams could challenge it, could come back to us and could tell us different things. It improved the data in, which improved our results coming out.
I think my crowning achievement on this, Emily, was that we flagged, using one of our language shift tone triggers, a problem that was emerging in packaging reforms. Everyone's really excited about making sure we don't mess up packaging reforms, but it was a director that had shifted their language and thought, actually, we had put this issue that no one was paying attention to through Cabinet subcommittees, and no one had picked it up. And then it was flagged through this process and would have caused billions of pounds of misspend across industries in changing packaging had it not been picked up.
The program team didn't feel they were being listened to. We flagged it up automatically. We asked, "Is this what you wanted?" And we were able to overcome that sort of issue.
Now, did we save anyone's life? No. But I think packaging reform was probably one of the clearest examples of, had we not had this, packaging reform was a small enough thing far enough down the agenda that it's highly unlikely it actually would have risen to the top of any other prioritization system. It wasn't the biggest program. It wasn't something that the press would've picked up on. But had it been messed up, and industry would have had outcry, then we would have been reacting to it after the fact.
Emily Lawson
So we built this system, mostly in 2022. In the late spring of 2023, I resigned from the delivery unit, and I left in September 2023. And what I thought I did on that exit was to set the system up for success.
I had this extraordinary team who was still there. I supported the Cabinet Secretary to appoint the next DG for delivery. I got the then DG of Public Spending in the Treasury to chair the delivery committee that was reviewing this algorithm.
But given what you probably can see on the slide, you will probably realize that that didn't go so well. So eight out of the 10 Colossus machines were dismantled after the war. So that initial extraordinary achievement was put into abeyance, and indeed, nobody could talk about it for a very long time.
But within that, there was a kernel of what had been produced and the people who had been through that experience, and in 1948, that generated the Manchester Baby.
And so, like our system, our approach was dismantled, but our team that produced it and even better tools live on. And Mallory has an example of somebody who's doing just that.
Mallory Durran
Yeah. So one of the people who had worked in the team developing the initial algorithms moved on to work for NISTA, which is what the Infrastructure and Projects Authority became. Because we had made so much of the code and the algorithm publicly available, they've picked that up and have made it better. They've brought large language models into the loop.
Now, it is still true that without strong leadership and a clear sponsor, not all of that work will rise to the top, and it may not be the single thing driving prioritization. But the work lives on, and there is still insight being generated from the spawn of what we created.
Emily Lawson
So we put together some thoughts about what we learned from this experience. Shall I make them all come up, and we can—
Mallory Durran
Yeah.
Emily Lawson
Rather than all of the builds. So if I start with this one, this might be one that—where's he gone?—Ben and I can debate about this, but our perspective would be that it is less critical for all the leaders in the room to be able to do the coding, although a nodding acquaintance is certainly no bad thing.
But what they do need to know is how good software is structured so that they know what to ask for. Because asking things that the teams have no hope of producing, or that they don't even have the right ontology and can't be interpreted, is indeed deeply unhelpful.
Mallory Durran
Yeah. And I think as a technologist, having a leader who appreciated what we could do to help bring us into that conversation was really important. But I didn't need Emily to know how to structure the training versus test data that I had, and the same thing will go for AI.
Emily doesn't need to know what the best harness is for the circumstances we're working in. She doesn't need to know what our scaling structure for the infrastructure looks like. I need my leadership to help me understand the context that I am building in.
You need to know what you are listening for, which sort of comes back to this. There is so much signal, and the faster we can scan the data, the faster we can come up with what we think an insight might be. But hypothesis testing is still a really important thing, and the faster you can scan an even bigger amount of information, the easier it is to get false signals or to not know what to do with the data that's coming up.
So starting from a really clear hypothesis, understanding the context, and defining systems that are based on that real content and context knowledge is as relevant now as ever.
Emily Lawson
We could both talk about this one.
Mallory Durran
Yeah.
Emily Lawson
I think the point—and I'll come back to this perhaps in a second in the close—but the thing that surprised both of us, I think, in the system is we wanted—I commissioned a system that would just give me a better answer, so I knew when to send people into a department and say, "I'm sorry, this help is not optional. This is really now a problem. You don't get to just say, 'No, thank you.'"
And what we actually got was a positive—we got that, but we got a positive feedback loop that was like, "Oh, actually, that packaging thing was really helpful. Maybe I actually need to put the actual concerns I have into the system, because then I might actually get some help."
And the system became a positively reinforcing loop of, if I tell the truth, I actually get better support, and we collectively get better answers. I'm not saying it worked everywhere for every department, but it certainly gave us a really strong foundation of trust between the center and departments, which is not always easy in government.
Mallory Durran
I think there was also leadership behavior in that because we didn't just flag up a signal and then haul people in to—
Emily Lawson
Yeah.
Mallory Durran
—rake them over a set of coals. And I think we started getting emails being like, "How do I get my thing up the agenda? Can you toss mine on there?"
And we stayed true to the algorithmic approach, and we said, "Give us your best quality data. We will check with you and tell you where you are on the list, and then we can decide together if that was accurate or not."
But the fact that people wanted to be at the top of that list because they wanted support is also a reflection of the leadership approach. The algorithm alone would not have achieved that.
Build it to outlast us. So I think this is about infrastructure. It's about openness. It's about sharing. The more people who knew what we were doing, the better the resilience of the technology was.
As we've said, it wasn't perfect. It didn't have an Emily to carry it forward. But had we not built a community, a network around it, brought people into our thinking, it would've done much less good. But now there are seeds of folks who worked in this way in every department across government who know what is possible and know they can work in this sort of way.
So invest in the infrastructure, make it sustainable, do it with the fewest number of people. I think things would probably be faster now. I think we pulled off the first version in four weeks—
Emily Lawson
Yeah.
Mallory Durran
—which is pretty good because we hate sleep and love government efficiency. But really just trying to be effective and put that sort of open source in place and build a community around it.
Emily Lawson
And so, as is compulsory in all Gene settings, we also thought of three areas of help that we thought people might be interested in. Mallory?
Mallory Durran
Yeah. So, stories of crossing the gap. Emily is my best example of how you do that, and I'm always telling people, "Find your Emily." But understanding whether there are less individual-dependency ways to go about that. How have you achieved that in your organizations? I think we would be really interested to hear.
If you've built something that is an early warning tool, can we add that to the catalog of practice that is available to people trying to do this sort of predictive forecasting? What will the world look like, maybe not in 10 years, but at least in sort of six months or a year, so that we can get ahead of it?
And what have you done to achieve life beyond that single sponsor? And so how do you make things sustainable when leadership changes, and especially in the public sector? Governments come and go like a breeze.
Emily Lawson
So our hypothesis would be that the machine is the easy part. And increasingly, I think that is the message that we've had today, that we heard in Charlotte, that most people in the room are experiencing every day.
The value is in knowing what to listen for and then being willing to act on what you hear. I am not going to go out and code the workforce model that I'm desperate for my team to code at the moment, or indeed to design the user interface in any one of our client policy tools. But I can be a leader who helps set those teams up for success, that is really interested in what they're telling me and is willing to hear the answers that they bring back, even if those are not what I originally wanted to hear or indeed that the organization wants to hear.
So my question, challenge out—or question, hopefully—out to the audience is: how are you creating the organizations that are asking the right questions, that do want to hear the solutions? And what's your response when these extraordinary AI tools bring us those answers more quickly, but in a way that perhaps we may not be ready to hear?
How do we get ready to hear that, and then what do we do when we've heard it? Thank you very much.
Mallory Durran
Thank you.
One membership. The IT Revolution library.
IT Revolution's books, the video library, the courses, and the MCP. The papers stay free.