AI Security Research in the UK Government

Host Intro (Gene Kim)

Okay. For those of you who, like me, were more than a little terrified at Nathan Labenz's presentation yesterday on the very real risks of AI going rogue, you probably caught more than a few references to the UK AI Security Institute.

As Lord Clement-Jones mentioned, this organization grew out of the first AI Safety Summit held here at Bletchley Park three years ago. Stuart Jennings is an engineering manager there and is responsible for the core platform and infrastructure for the institute, including the sandboxes where the most capable AI models are evaluated.

Life must be exciting. We were talking earlier today with Nathan, Stuart and I, and we were talking about a book that is often considered required reading among the early people in AI safety, and that book's name is called The Making of the Atomic Bomb. It chronicles a story that is full of near misses, people doing extraordinary engineering at the very edge of what anyone understood, in a race, but also trying to avoid catastrophe.

So before AISI, Stuart spent eight years at the BBC, where he founded and scaled the team that built AI tools for journalists. Please welcome Stuart.

Stuart Jennings

Hi, everyone. It's really exciting to be here.

So I am going to talk to you about the UK AI Security Institute. And we are an AI security research institute within the UK government. I'm going to talk through what AISI, as we call ourselves, is, a bit about our research and the sorts of risks that we focus on, and what it means for you, focusing particularly on cyber risks.

I'm Stuart Jennings. As Gene said, I won't introduce myself again.

So firstly, for my context, who here has heard of AISI? Loads of people. Okay. Thank you, Nathan, for name-dropping us yesterday.

So firstly, what is AISI? In one sentence, we equip government with a scientific understanding of the risks posed by advanced AI.

But where we started was a small group of civil servants set up initially as the Frontier AI Taskforce. Our main purpose was to try and set up this big AI Safety Summit, which happened here, eventually, so it's very cool to be here. That happened in 2023, and then, shortly afterwards, the AI Safety Institute, as it was called then, was founded, and then we changed our name to AI Security Institute.

This was a bold new idea. We were really the first institute of our kind. It was also seen as a slightly mad idea, setting up a group of highly credible technical experts within the government. But since we've been created, many other countries—I think the count is about nine now—have set up equivalent organizations.

And the idea—this is a bit of a cliché, but I think in AISI it really holds true—is that we would be like a startup within government. We were given lots of funding, 66 million a year at the time. This has now grown. And we really had a lot of direction from the Prime Minister's level to cut red tape and give us space to grow and move very quickly.

We've now grown into an organization of 250 people in total, with more than 100 world-class technical staff, with alumni from places like Anthropic, OpenAI, Google DeepMind, top universities, other parts of the civil service. And we are, I believe, the largest and most sophisticated AI safety organization in the world.

But what do we do?

So our bread-and-butter work is testing and evaluating frontier AI models. We have special arrangements with the frontier AI companies, and they give us early access before release to these models, so that we can run our suite of evaluations that measure it across various different dangerous capabilities.

We then feed this information back to the AI companies. They can improve their safety practices before release. It can impact their release decisions, which makes everyone safer.

We also then inform policymakers. So having this early access also gives the UK government and allied governments advance notice about what capabilities are coming. We produce lots of reports based on broader AI trends and our own research, so that policymakers have the most up-to-date, technically credible advice on AI.

We also do lots of our own research in-house. So we publish a lot to advance the state of AI safety in general, but also seek to directly positively influence AI safety practices at the top AI companies.

We have various different research teams, and our mission is to focus on the most severe risks.

So, for example, we have a cyber and autonomous systems team, which seeks to measure and understand AI cyber capability, and also ability to carry out autonomous tasks. So they do things like mapping out various different scenarios where humans might lose control of AI.

We have a chem-bio team, which looks at measuring AI capability to create things like novel pathogens that could cause new pandemics.

Our human influence team looks at how AI can influence humans and society at large.

Our red team is the best in the world at stress-testing the safety features of these models, often before they're released, so that the AI companies can strengthen them before they're released to the wider public.

And our model transparency team does interpretability research to see what's happening under the hood of these AI models, and also examine their outputs, their chain of thought, to understand their intentions and build concrete mitigations to control and monitor the models.

And my team, the engineering team, builds a lot of the platform tools that underpin all of this. As Gene mentioned, we build the sandboxes that try and contain these agents. We do a lot of the security engineering. And we've also notably built the Inspect framework, which is now the global standard framework for anyone evaluating AI models.

So I'm now going to talk through some of the risks that we focus on and a selection of our research.

So, as I mentioned, our red team exists to try and break the safeguards that models have. So places like OpenAI and Anthropic will build safeguards that stop people just saying, “How can I make a bomb?” for example. You have probably encountered these yourself in using AI models.

Initially, these were quite easy to bypass, but these safeguards have got more and more sophisticated. But there are always bad actors out in the world who will seek to try and get past these safeguards, essentially nullifying them and giving them access to these powerful models for bad ends.

One notable success of our red team was to develop what we call the universal jailbreak. This means an algorithm that, once supplied to a frontier AI model, bypasses safeguards on a wide range of dangerous capabilities.

So in this case, this specific frontier model would previously answer 0% of a set of dangerous biology questions, and after this jailbreak was submitted, it would answer 68% of them. We also found that this attack cost about $330 to develop, so it's very widely available.

But luckily, our red team found this before the model was released, and both OpenAI and Anthropic fixed the model, making it safer for everyone.

Another really notable and quite scary capability of AI is in uplifting bad actors to create bioweapons.

So the scientific community will publish information about the tools that they use to make things like viruses or medicines, but they won't publish details about the tools themselves. They'll just publish a scientific paper describing the tool in general.

And these tools are dual purpose. They can be used for good or bad ends. But our chem-bio team found that if you simply point a frontier AI system at some of these papers, it can reconstruct the tool anyway. And again, cost about $20 to do.

We also found that AI can sway political opinions. So this article that we published in Science shows that AI is very capable of changing people's political views after just a short conversation.

So we conducted a study across 76,000 participants and 700 political issues and found that cheap, open-weight models that had been deliberately trained to be persuasive often outperformed larger frontier models. And we also found that the most persuasive arguments tended to also be the least accurate.

So again, this shows that bad actors could fairly straightforwardly deploy large fleets of these cheap open-weight models deliberately trying to be persuasive to, for example, sway an election result in a democracy.

I'd like to focus particularly on cyber risks. Obviously, this has been in the news a lot recently, and it feels like this year is the year where a lot of risks that the AI safety community have been talking about for a while in cyber really became real.

So we published research earlier in the year that was an early indication of incidents that have happened over the summer. We found that AI cyber capability, measured on hacking challenges that we've developed, was doubling roughly every 4.7 months, which is a rate of increase that's kind of hard to internalize.

We found that up to 12 hours of equivalent cyber expert work carried out completely autonomously was within reach. And we'll shortly be publishing an update to this with newer models, but we found that this rate of increase was faster than we estimated a year ago.

So you can see on this graph, this is a log scale, so this is an exponential increase. But you can see that Claude Mythos Preview and GPT-5.5 were above this curve. It was a super-exponential increase.

We also found in April several other early indications of cyber capability. So we have these things called cyber ranges, which are essentially large, complicated cyber assault courses for AI systems to solve. They aim to fully simulate a real company's network.

Previously, the most capable models had solved only about three quarters to two thirds of these tasks, and they involve up to 32 steps autonomously carried out in one session. But in April, one model completely finished this end to end.

We also published some research in March with a new benchmark that we developed measuring AI capability at breaking out of their sandbox environments. So you can see illustrated in the diagram to the right, basically the evaluation involves a model prompted to try and escape from a sandbox where we have deliberately implanted a vulnerability, and there's a range of tasks that are increasingly difficult to carry out.

In March, the most capable model could only solve about 56% of these. But weeks after we published the benchmark, Claude Mythos Preview saturated the whole thing and solved 100% of them.

We also found that every single model we tested cheated on its tasks.

So we found that across a range of frontier models on our cyber evaluations, eight to 14% of the runs, the model attempted to cheat. So this involves going beyond the bounds of its evaluation to achieve its goal in some unexpected way. This could be relatively benign or could involve hacking into another company.

We also found that the models didn't reliably report when asked if they had cheated. So we followed up and said, “Did you cheat?” And most of them said no. And under half of them said it was wrong to have done that.

So, timeline. Yes, earlier in the year, we had these results showing these rapidly increasing cyber capabilities. In July, we had the famous OpenAI and Hugging Face attack. I won't go into this. I assume most people are familiar.

If you're not familiar, I highly recommend reading some of the detailed reports. It's quite scary reading, but essentially involves a fleet of thousands of agents coordinating, breaking out of their sandbox environments, accessing the internet and hacking into another organization.

We also had our own incident at around the same time, which was different. It wasn't a sandbox escape. It involved an agent that had been deliberately given access to the internet, but where, as part of a cyber evaluation, it tried to merge some real code into a real open-source piece of software and created multiple fake GitHub accounts to try and socially engineer the maintainer of that code to accept it. And we think this was the first major in-the-wild example of social engineering.

And more recently, big news like PHP Shell has shown the widespread impact of these sorts of AI vulnerabilities and the need for people to act very quickly.

AISI's research has also shown that open-weight models are only four to seven months behind the closed-weight frontier. So these open-weight models are free to download, and this gap has shrunk from six to 10 months last year.

And these models can be run at a fraction of the cost and often have fewer safeguards to begin with when they're downloaded, but also these safeguards can be relatively trivially trained out of the model, and people can distribute a totally new version of the model that has maximum capabilities and minimum safeguards.

So again, this means that these sorts of frontier cyber capabilities are only lagging four months before models that are going to be widely democratically available to people with no safeguards.

So what does this mean for you?

It's not AISI's job to directly give advice on cybersecurity. That's what the NCSC is for. However, as a minimum, please sign up to the NCSC's Early Warning scheme. It will give you tailored updates to your website about vulnerabilities that are relevant to you.

And also, they have a scheme called Cyber Essentials. This is an accreditation scheme where you can talk to them, get signed up, and they will accredit you to say that you have met their minimum basic standards for cybersecurity. So please encourage your organizations to do this.

But to give a flavor, this is how we do our own defense.

So when we are running these models, we have multiple layers of defense, and we assume that any single one of them can fail.

So we will run the agent in an isolated sandbox environment with only the tools that the task needs. Automatic checks before each run to make sure that task boundaries are explicit, the safeguards are in place, and two layers of isolation around everything. Two layers of network isolation, two layers of host isolation.

And lots and lots of logs, transcripts, all in one place linked to alerts and automatic kill switches when any suspicious activity happens. And also, crucially, using AI to defend on basically every level.

We have an LLM that reviews the tool calls of the agents that we're testing and can block anything suspicious automatically. We have LLMs that look at a broad range of transcripts generated by a fleet of agents to see if anything suspicious happens, also comparing it to network logs.

And we also use AI very proactively to scan the sandbox environments that we run our agents in for vulnerabilities, as well as using AI to look at the external surface of our organization to protect against external attacks.

So to finish, three asks.

One, please spread the word. Talk about AISI, talk about AI risks. Talk about cyber, but also the broader range of AI risks. These are all real, and they're going to be affecting society increasingly in the coming months.

And also, please tell me what risks you're seeing. I'm very interested to hear if you have experienced AI-enabled cyber attacks yourself.

And please follow NCSC guidance as a minimum.

Thank you very much.

Gene Kim

Okay. Thank you. Thank you, Stuart.