Inspect the House, Not the Neighborhood: What Actually Makes an AI Model Risky

Host Intro (Gene Kim)

All right. Philip mentioned where AI destroyed everything. I think I need to share with you a story of where I did something where AI destroyed everything. So earlier this year, I was in Salt Lake City. We were doing this live coding workshop at the Ship Summit, which is where I met Kevin, who you heard from earlier this morning.

And I asked Cloud Code to move my VM in Google Cloud from US Central to US West 1, which is closer to Salt Lake City, to make it a little faster and a little nicer for everyone who's going to be in the workshop. Which Cloud Code did, but only after it deleted all the VMs and the storage buckets that had all the backup files and the disks.

So this was like T-90 minutes before the workshop. I managed to get it running for one workshop, and I was worried how I was going to get the second one running. And that was when I was happy to accept the help of Nicky Pike, Field CTO of Coder. They already had a kickass environment that they were using for some of their activities, already wired into a CI/CD pipeline, all on the Coder platform. And so how Nicky found himself there is maybe no surprise. He spent years at Pivotal Labs. He spanned operations, infrastructure, development. He hosts the [Dev]olution podcast, and we had this hilarious conversation about open weight models. And he had a hilarious and jaundiced and maybe a little cynical view of the views, myths, worries, and fretting that we have on open source models. And when I heard that, I immediately thought, "You should definitely be presenting here." So here's Nicky.

Nicky Pike

All right. In May, the hottest model on Hugging Face — it was a fake. It showed up as the official OpenAI release, had the same model name, posted from a lookalike account with OpenAI's model card copied word for word. Inside of 18 hours, it was the number one trending model on the site. It had about 244,000 downloads, and the researchers who caught it, they think that bots were used to pump those numbers up. And it worked. Popular made the model look vetted.

But in this model, the malware wasn't hidden within the model. It was in a setup script that was with a README that you were told to run. This attack was specifically made for Windows. You run the script there, and it went through the machine for saved passwords, browser sessions, and crypto wallets, and then it shipped all that information out.

Now, think about the question that a lot of security teams were asking when this happened. Where did this model come from? This one had OpenAI on the label. If you have a policy that only checks where a model comes from, chances are that this one would've gotten waved right through. The damage came from what people ran once it was on their machine.

Inside most companies, "where did this model come from" shows up as one very specific question.

Quick show of hands. Who has been asked in the last six months, "Are we using any Chinese AI?" Whether it be from your customers, your board, or a security questionnaire.

All right. I'm guessing a lot of you would've probably answered, "No, we're not allowed to use Chinese models." Well, raise your hand if you had a clean answer to why you weren't allowed to use those models. And I want something specific. I don't want "it could've had a backdoor" or "it might phone home." Something that the model actually did. Now, public sector, I know your answer. For you guys, the law says no. That's a real reason, and we're going to get into the answer a little bit later. But for everyone else, if you noticed, not a lot of hands went up. There's a lot of fear about where a model is born, but very little evidence about where it being born being the problem.

So why does this question keep coming up?

Well, it's the bill. Frontier models cost real money, and the invoice is growing a lot faster than anybody planned, and that invoice is starting to sting a little bit. So when that happens, the next idea is always to put up an open weight model, the kind that you can download and you can run yourself on the routine work that you have. And right now, most of the best open weight models, they're coming out of Chinese labs. We're talking Qwen, DeepSeek, GLM, Kimi. So the cheap option and the scary option turn out to be the same option.

Now, before we get into the scary — are open weights really cheaper? Well, I've got three stories for you from the last year. These stories don't agree with each other, and I think that's kind of useful.

First, the case for paying up. A quantitative trading firm that we're working with, they watched one engineer run up a very expensive bill in about two weeks. Then they saw what it bought him. He had built a hardware simulation platform that they had planned as a 12-month project costing over a million dollars. This engineer, he finished it in two weeks, and the token cost that they had for that was a fraction of the million that they had planned. Now, when we were talking this through, one of their folks joked that the easiest way to spend company money without asking anybody used to be Lambda functions. But right now, tokens are putting up a good fight, and they're giving Lambda functions a run for their money. For them, that's paid off, but nobody had to approve it first, though. That is both the joke — it's also the risk.

On the other side, a large logistics company, they went the other way with this. An engineer put Gemma on a couple of cloud GPU clusters, and they wanted to see if self-hosting would save them money. It worked just fine, but then he did the math. At the usage they had, the GPU nodes ate into the savings that they were expecting. So it might pay off at a bigger scale, but it wasn't for them today, so they shelved the project.

And one of ours. At Coder, we built a support agent that we lovingly call Blink, and we put it into our customer Slack channels. Now, it runs on Claude Opus, which is one of the frontier models, but that agent — one customer that we had opened more than 300 conversations with it over the last 12 months. Two of the three were real back-and-forth conversations, averaging about 25 messages per thread. Without Blink, those would've been support tickets, or they would've been questions that the customer wouldn't have even asked us. Total token cost for that customer for the year came in at about $300 for us.

So of those three stories, we got one that was expensive and worth it, another that was self-hosted but wasn't cheaper for them, and we've got a frontier model for a full year under 300 bucks.

So is open source cheaper? Sometimes. You have to price your workload on your requirements, and you have to tier the models. You put the big models where they go to earn their keep, and the open weight goes where the volume is high and the stakes are low.

Now, the minute that you try open weight, somebody asks the question, right? This is one from the show of hands. That same logistics company, they also tried open weight models from other labs. Security took a look at the country of origin, and in their engineering words, cybersecurity was like, "Absolutely not." And I do think that that instinct makes sense. I just think that it's aimed at part of the problem that's the easiest to see. And I want to explain this in a way that I think is useful — and in a comparison that's somewhat personal to me.

When people go to buy a house, they start with the neighborhood. Is it a good zip code? Are the school districts good? Who's going to live next door to me? All of that is worth knowing, but none of it tells you what's behind the drywall in the house that you're wanting to buy.

My wife and I, we didn't buy our house. We built it ourselves with our own hands, and we did it mostly off of YouTube tutorials. We lovingly refer to this house right here as the house that YouTube built. So I know what's in those walls because I'm the one that put it there. But guess what? I still find things that I got wrong.

The point is, if the people that built the walls can't be sure what's in them, then you can't be sure about a model that somebody else has trained. And that's true no matter where the model was born.

So think of a model as a house with three layers. You've got the wiring — what got baked in before you ever moved into it. The contractor — who built it, and what they left behind when they handed you the keys, and do you even trust them? And the doors — what can get in and what can get out?

Right now, the AI conversation is almost all about the neighborhood, where the model was trained and whose flag is on the building. Very little of it is about how the house was built. So let's walk the house.

We're going to start in the walls. The wiring is whatever got trained into the weights. It's also where the fear behind the Chinese AI question really lives. That fear is that there's something hidden inside that model.

In January of 2024, Anthropic published a study that they called Sleeper Agents. They trained a model to write safe code when the prompt said the year was 2023, and code with a hidden hole in it when the prompt said 2024. Then they ran standard safety testing against that model to try to get the backdoor out. It stayed. And in some cases, that training even taught the model to hide the backdoor better. Now, what this means is that safety testing could make a team feel confident in a model, but there could still be a problem in it.

Last October, Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that about 250 poison documents could plant a backdoor in a model, and that number held across every model size that they tested. The old assumption was that the bigger the model, the more data that it was trained on, the harder it would be to poison. Their study proved that that wasn't true.

Now, two caveats on that. Both of those backdoors, they were built on purpose in a lab, and the poisoning one was very low stakes. It made the model basically just spit out gibberish. So I went looking, and I haven't been able to find a single public case of a shipped commercial model that was carrying a backdoor, American or otherwise, and that's the evidence gap from what we got from the show of hands.

Now, there are research scanners out there today. Microsoft published one this year that caught most of the poison models that they tested, according to them. But folks, most isn't all. There's nothing that can tell you that a set of weights is clean. If you can't fully inspect the wiring, then you have to plan around it, and we're going to get to how.

If the wiring is the risk that nobody's caught in the wild, then the contractor's the one that's actually already hurt people. A lot of models on public hubs are saved in a Python format called Pickle. And opening a Pickle file can run code. If you load one of these the wrong way, then it is the same as taking in and running code that you got from a stranger. In 2024, JFrog went out and they found that about 100 models on Hugging Face had working malicious payloads, and nearly all of them were hidden in this way.

That fake OpenAI repo — that's a contractor story, too. The malware was what they left behind, a script that was sitting next to the model with instructions to run it. And where does that script usually run? On whatever machine somebody used to download it. And they wanted to try it out. Often, that's a developer's laptop. That's their machine, something where they're logged into everything. One bad file there hands over whatever that developer can reach. But notice — the model never did anything. It was the files that did it.

An investment firm that I work with, they sidestep this. Nobody there is allowed to run models locally, and that's partly by policy. It's partly because, in their words, "Development laptops have the resources of a pocket calculator for running AI." They want to run their stuff on servers. For them, a model file never lands next to a developer's keys, and I think that's the right way. Now, most of you probably couldn't make that rule overnight, much less try to enforce it.

So if you do want to pull from a public hub, here's three habits for you. Use safetensors. Safetensors — that's just the numbers, and it can't run code. And you pin that model to a commit, an exact commit instead of latest. A repo can change after you've checked it, and pinning the model means that what you've tested is what you're running. And make sure you leave `trust_remote_code` off so that the library doesn't try to run the repo's Python itself.

Now the doors. This is the other big worry. Does my data leave?

For a hosted service, that's a fair question. DeepSeek's own privacy policy says that when it's hosted, the data that it collects is stored on servers in China. That same investment firm that I was talking about — they told us that this was the real concern for them. Their answer to that was approved models from approved providers through an approved AI gateway. Nothing talks to the internet unless somebody approves it, and that includes the remote MCP servers, the tools that the agents actually plug into. That one rule protects them whether the model came from China or California or anywhere else, and the way that they enabled this is through self-hosting.

Self-hosting gives you a chance to architect out the fear from these models. It gives you the chance, but it doesn't fix it for you. If you stand a model up with an agent that has open internet access, your data can still walk right through it. The work to fix that is yours. You block outbound by default, allow what you've approved, and you watch what tries to leave. With a hosted vendor, you don't get the chance to do that. You get the controls they offer and a promise from them that they're going to do their best to keep you safe.

And doors are not only about the data leaving. They're also about what a model's allowed to do, and a model doesn't do anything by itself. It sends instructions, and an agent carries them out. Remember, you can't see the wiring of the model. If a model does have something bad baked into it, you can only act through the agent, and the agent can only use the doors that you give it.

We learned this on our own agent, and we didn't even have a backdoor in it. We used to let Blink open pull requests straight from our customer channels. What we found was that that caused too much low-quality code that was landing on our engineers, and it was distracting them. So we turned that off. Blink didn't need that door, so we closed it, and we locked it. Notice that none of that cared about where the model came from.

So where do bans fit?

Well, banning a hosted service that ships your data overseas — perfectly reasonable, right? And if your biggest customer's contract says no Chinese models, you drop them. That's a business call, and it's a fair one. Just be very clear with your board about what that buys you. A poisoned American model loads the exact same way, and that fake repo had an American name on it.

And also remember, the list of what's getting banned changes. This year, the Department of War labeled an American company a supply chain risk for the very first time. That was Anthropic. Defense contractors started getting requests from different agencies with different deadlines and different scopes to find and certify where they use Claude. The designation itself is still being fought in court. I'm not here to say who I think is right or wrong, but I do want to use that example as a reason to ask you one question.

If that letter landed on your desk on Monday, could you tell me on Friday everywhere that model runs in your company?

From what I see, most companies can't. I was on a call with a consumer software company where one of their people mentioned they were trying out a new model through Bedrock. His colleague on the same call said, "I had no idea we were even trying that." That's the Friday question failing in real time inside one team.

And finding it is only half the problem. The other half is being able to switch it out or stop it. A large financial services firm had a couple of models go bad over a backend change one weekend. Afterward, the ask from them was very simple. If Model A is not available, then route it to Model B. They didn't have that switch when they needed it, and now it is a critical requirement for them going forward.

That switch is the answer to the wiring problem that we talked about earlier. You can't fully inspect the wiring, so you have to plan for it.

So let's zoom out a little bit. We've been inspecting the model like it's a house. Inside your company, the model is just one appliance inside of a much bigger house. Every water heater dies. You know that when you install it. That's why we don't weld it to the wall, and we don't pour concrete around it. The model is the water heater here. It will get replaced, whether it gets poisoned, it gets banned, or somebody simply ships a better one.

So here's how we build for that at Coder. Our engineers use Cloud Code, use Codex, and they use our own Coder agents. That list changes for us every quarter, and the models underneath it, they change with it. What stays the same for us is the way that every agent works in the same kind of workspace on our infrastructure, and our model calls go through one gateway.

That one choke point is what makes the rest possible. We can see every call, who made it, which model answered, and the tool calls that it made. We're able to set monthly spend limits per person at that same location so that every token bill is a decision that somebody made, and it's not something that we're surprising our CFO with.

So what does a model swap actually take? Well, we're talking about houses, so let's stick with that. Think about how a good plumber installs the water heater. They put a shutoff valve on the water line and flexible connectors instead of pipe soldered straight in. You close the valve, you unhook the water heater, you drop a new one in.

Same idea here. Your shutoff valves: a policy at the source, either at your cloud account or on your own model servers, that blocks a model family for every tool; and a gateway where you can shut off the provider and see who's using it. Your connectors are configs that point each tool at that model. For us, that's our workspace templates. We point the configs at the workflow, and we use the replacement. If we want to replace, we change the config and we push that out. No laptop's touched, and if you stay within the same API format, nobody has to rewrite a workflow. They may not even know that it happened.

Let the model be the part that you replace. Let it be an implementation detail that's written into a config. You keep the rules about who can use what, where, and what it can reach. And for everything that runs through the gateway, that Friday question now becomes a simple lookup.

So let me leave you with three questions for every model that you want to bring in.

Wiring: Can you verify what's baked into that model? No, not fully. So you have to build your infrastructure for the swap.

Contractor: Do you control how it loads? If a file can run like code, you should treat it like code.

Doors: What can get in and what can get out? Lock out everything that it doesn't need.

Here's your homework. Pick one model that your company uses and try to answer the Friday question for it this week. How long it takes you — that'll tell you where you stand. Get that answer down to minutes, and the next time somebody asks, "Are we using any Chinese AI?" instead of arguing about where it was born or if it's safe, you can have a clean answer. You can say what that model can reach and how fast you can pull it out.

And if your answer takes longer than a few minutes, it's okay. You're in good company. A hedge fund that I talked to, they want to run exactly the mix that we started at the top of this talk. They want open weight models on their own GPUs for the high volume, low impact work and hosted models through their cloud. Their words: "Figuring out what that hybrid looks like, we're still not there yet." I hear that from a lot of teams, and if you figured out that mix, please come find me afterwards because I probably want to steal it from you.

So let me end where I started. The people who ran the fake repo, they trusted a label, and that label said OpenAI. Look where you can, lock your doors, and build so that what's behind the walls has the ability to come out easy. Thank you everyone.

Please feel free to come see me at the Coder booth out in the hall if you have any questions or you want to talk more. Appreciate you.