Adversarial Code Assurance as a Way Back to the Paved Road

Host Intro (Gene Kim)

Our next speaker is James Wickett, who I met in 2012 in Austin, Texas, at the South by Southwest conference. He's one of the three people who can legitimately say they put the "sec" in DevSecOps. In other words, he was one of the first people who seriously talked about how do we integrate information security objectives into the daily work of everyone, especially developers.

In fact, when we wrote The DevOps Handbook, the information security chapter started with citing some of James' work extensively. And so he's one of the best boundary spanners I've ever met. He's been in almost every functional role, and in my mind, he was near the epicenter, if not at the epicenter, of almost every interesting event in the DevOps movement. And in 2009, he was a part of a group at National Instruments that built the hosted LabVIEW, their first cloud product. And how they brought that to market was actually how we highlighted a greenfield project in The DevOps Handbook.

Anyway, so he continues to live on the frontier. He's now CEO and co-founder of DryRun Security, whose work was cited by Randy and Michelle yesterday. And so they use AI to do security reviews of code, millions of lines a year. And so when we looked at the program, we saw a gap in information security, and I couldn't think of a better person to invite than James.

James Wickett

Hey, everybody. Oh, should we play the music? I like the music choice. It's great.

Well, happy National Cybersecurity Month to you, right? I don't know, what do we say about that? It's October, right? This is a thing. That's about the level of the jokes, so that's where we're going to be. Hope that you're okay with that.

So, I really appreciate Gene's intro. He was very, very kind there.

I work at DryRun Security. We do adversarial code reviews so you can get AI code assurance. That's the whole point. And really, this talk is sort of a way to explore — all right, I want to put adversarial code, from a security perspective, analysis in my pipeline, or how I'm doing my AI SDLC — sorry, or your SDLC, or whatever you want to call it. Names are things, whatever. But what do we do? What do we do now?

And one of the ways we want to think about it is, how do we deal with the current environment that we're dealing with? I really liked Christina Yakomin's talk yesterday about security being kind of that next layer of where people are trying to solve and trying to get to.

So, life's been going really good for me. I'm really good at Vim these days, like all those years. Anyway. Thank you, for whoever laughed — it was very kind of you.

But for the agentic AI and code reviews paper, we sort of covered some of the main things that need to happen, because in the words of a lot of developers these days, it's like, "I'm not going to read that pull request. I'm not going to read that code." It's like it's unreadable now, is what some people have said. And in that, we talk about a few items.

One of them is the idea of adversarial code review. We talk about having deterministic engineering baked in at different spots. Being able to deal with context that's a part of the code that's been making its way into your product, or making its way into the pull request or the code base as it's changing. And we were also really talking a lot in that paper about doing multi-agent reviews and having multiple adversaries make those assessments.

And so what's going on in the world of AppSec right now is we're seeing that AI software factories — and you can have a loose definition of that however you would want — is breaking current application security.

So, a lot of the PRs — the first number here, 87% — we set three different coding agents on some coding tasks, and we found that in 87% of the PRs, they had some security problem in them to some degree. Now, sometimes they would naturally, over the course of doing several tasks in a row, solve some of the previous existing stuff. So you don't always make a one-to-one mapping on that, but maybe they've introduced something new or in a different area. And that was some testing we did back in March.

Okay, the 5X code volume — that's just sort of my guess. I don't know. I mean, everybody's kind of seeing this trend just going up. I'm going to share some real data of what we've been seeing. But I think the most concerning factor in AppSec and the way that people are thinking about application security right now is that most of the security vulnerabilities that the AI coding agents are making are undetectable by traditional means.

So they don't really fit in your neat cross-site scripting or SQL injection buckets — although I've certainly had that happen. I've had some cross-site scripting by an agent before — but it is more like IDOR, BOLA, broken auth, authorization RBAC, that kind of stuff. Logic stuff.

And I think one of the things that we're all wrestling with, and we're dealing with this at the conference — I really, really enjoy a lot of the content here — and I don't know if any of us would argue with this statement, but all this demand for generation is creating the same proportional need for verification. Verification is kind of the name of the game right now, and many of us in our organizations are trying to sort out: okay, how do we prove this thing? Or how do we set the bar of — this is okay to merge, or this is the thing that should be allowed in?

Many of us find ourselves in a situation where AI code generation is like the Wild West right now. You can go to anybody, you can ask them, "Do you have any AI code?" I remember, like a year ago, some people would be like, "No, we block it here." And I'm like, "Yeah, okay. Good story." But then nowadays it's like, you can be like, "Yep, we're an all-Claude shop." And then you go look at their stuff, and it's like, no, they've got pretty much everything, right? Everybody has everything in a lot of organizations.

And a lot of the application security folks are still trying to apply kind of old tools or older methodologies to do the detection, and then trying to figure out ways to bootstrap some AI agents to do some of the code reviews and put that in, but it's still nascent in the AppSec space, I would say.

In 2013, Jason Chan gave this really great talk, and he was talking about the paved road, and it was really one of those things that in those early days of DevOps and security and Rugged DevOps and DevSecOps, we're just all trying to find our way. Netflix was here, like, okay, here's how we're going to provide this paved road, which kind of came out of — I learned recently — came out of the work at Pivotal Labs, which I guess everything comes from Pivotal.

But anyway, so this is moving from gates to guardrails, and he was really kind of making this argument that you want to make it easy for developers to participate. And I think this is an old concept, but many of us have been familiar with it.

Now, I don't really like shift left as it was applied to security. If you ever hear about a developer talking about security, how they shifted something left, they say something that rhymes with "shift," and they keep the "left" part — but they don't really feel like security did the right thing there, right? They just took the same tools, processes, and dumped the output on developers. And so then they're like, okay, now my Jira backlog's 5,000 tickets, or 2,000 dependencies, or whatever it has to do.

So Jason's comments on the paved road always resonated with me because I was like, all right, that's what we really need to do. We need to rethink what we're doing, because even back then, as we were moving from assessments and we're doing 10 deploys a day, 50 deploys a week, or whatever it was from the organizations — it's like, how do we allow that to go faster and still have security in mind and in view? And the way you do that is through more of a paved road, instead of trying to take the intellectual burden or dealing with all the output from these security ticketing systems or eventing systems and putting it onto the developers.

So, okay, I want to walk through a real scenario, and then we'll kind of walk through — all right, what do we do about this?

So, we tell our coding agent we want Alice to be able to view an invoice in our system. And so it's like, okay, cool. And it goes off and it writes some code. And you're like, okay, this code looks good, I guess. Can anybody spot the problem with this code? Yeah. Okay, probably somebody's got it, but it's okay. But it's like — okay, it doesn't really specify what tenant ID, right? It's like, you're saying, okay, I know my invoice ID, and it's probably some random number, random value, but Alice works at company A, she could see company B's stuff, at least from the code that we have here.

And so we're like, "Hey, Claude, not good. Alice can read the invoice from other tenants." And there's a really funny video — if you get the slides, you should watch this video. But it's about Claude, and anyways, it'll make you feel good about your life right now.

But okay, so Claude's like, "All right, I made a mistake. I'm going to go ahead and fix that." And then it's like, oh, I'll set it in our HTTP headers, and I'll pull that out from whatever that value I have. But the problem is that environment variable can be manipulated, so not good still. Alice could still read it if you were to fiddle with that value.

And then the actual repair is: hey, check the tenant ID when you're looking at it. And so that's classic IDOR — insecure direct object reference. Humans write it. Legacy tools, like pattern matching tools, can't really find it. You really need a machine to reason about it, or an AI system to reason about it and think through how this is executed and kind of walk the path.

So, at the beginning, I said AI problems are different than regular problems, and so let's go through a little bit of data. And so I thought I would pull in some AI success metrics here. And I'm going to start with the most surprising of all.

So I looked through all of DryRun's customer data and selected some at random, and I found out something really surprising. This is going to blow you away. You might want to take a picture of this one: 100% AI coding adoption. Everybody's doing it. You probably didn't guess that, right? Okay. I told you that's where the humor is here.

So, we're a young company — we started in 2023. But we only tracked the repos for this slide that have been active with us for six quarters, and we've already seen just in those existing repos almost a 2X gain in pull requests and activity. And so that's been really interesting.

We're also looking at organizations that exhibit what we are just kind of defining as software factory activity. They're picking up the ticket, they're writing some code, and they're committing the change. And the percentage of organizations across all of our customers and open source users — we've seen now almost 30% have some software factory activity. Is it everywhere? Is it spread evenly? No. But it is happening, and we're seeing this trend continue.

It frustrates my co-founder because he's like, "I didn't know. I didn't think we'd see so much adoption of this." And I'm like, "Yeah, I think we're definitely going to continue to see more and more of this."

Also, we saw that one out of five PRs changes security posture. And so I think it's like we can think, our threat model's good. Well, your threat model's out of date by Tuesday, right? Or the wiki page that we have, or whatever it is that we think we have an idea of what's going on in our system. And these are things like — it's changing our CI/CD pipeline, it's changing RBAC, it's changing GraphQL schema, adding a new API endpoint. And then we also analyzed our data where out of 13 pull requests or 13 code merges, one out of 13 is adding new attack surface that we think would be credible. That's not to say they actually have a vulnerability in it, but they're expanding in areas that are really, really important for us to think about when we do this. And I'll get into why we consider that.

Also, people are baking in a lot of AI dependencies in their code already. So we're seeing people putting in SDKs for LLMs, putting in MCPs. Now, I know that some of these are self-selected users who are using DryRun Security, so I know that Nicole Forsgren would not approve of my selection method for the data. But maybe we'll continue to work on getting that better.

So I also look at this and we think about what are the AI-specific risks that are happening and where the more classic risks fit. You'll notice cross-site scripting, SQL injection, that sort of stuff, it's towards the bottom. But IDOR, number one; prompt injection, number two. And those are consistently — we're seeing a lot of that activity happen just across the entire code base. We're doing around 600,000 code reviews a month right now. And I see us probably getting to a million by November, December.

But what does it look like? I mean, we can think — okay, so that's the data side of it. But what does it really look like to do an adversarial review, and what does that really mean when you're trying to challenge the code change and you're thinking about it?

One is you've got to understand the system. So we want to know code history, context. You want to know your security boundaries. You want to challenge the code independently. So you want to have — not just a different agent that wrote the code, like a Claude wrote it and Codex reviews it. Some people do that. That is an okay way to do it to start. But you want to come from a different frame of reference. So either wrapping in a harness or wrapping different tools it has tool access to that can do other investigations — those are, I think, really valid ways to do that sort of adversarial review.

And then exploit chaining is another one. We've all seen Mythics, and we see the power of that. But even in our code base, from an AppSec perspective, if you're able to put a few of these together — two or three lows or mediums — they might be workable as a high, right? And you want to know: okay, should we fix this or not? Because many teams are still kind of stuck on that. Well, we fix criticals and highs, and everything else, we just don't deal with.

You want to understand your security contract — like, what's the normal stuff? And I think security has this really interesting problem because we're not just testing explicit conditions in testing like we would for unit testing or BDD or something like that. We're testing implicit industry-wide components as well, thinking, how could this bit be attacked? And it is trying to solve that kind of unknown-unknown problem. But we could at least say for our organization, you've got to pass this level of checks. And then there are other things that we know are important to us, and we want to have our own policies and rules to put in place — that's our security contract to our customers, to the other folks in our organization.

And then any suggested tool that gives you feedback — you want to have that feedback loop where it can happen automatically. So it's not: create a ticket, or create a merge, and then that goes and creates a ticket, and then somebody gets to it. You want that to happen right at merge time, or before merge time. You want it to — it goes in, gets merged, it finds a problem, and then the remediation flows there.

Okay, I'll say one thing: I think that harnesses are cool. That's great. But don't over-index on a harness, because a harness misses a lot of components, and it still kind of leaves a black box of thinking over into the LLM itself. It really hinges on how good you are at giving it the prompts and skills and other components there, and then tool access. And you are sometimes still kind of reliant on the underlying LLM that you're putting in place. And so you'll have different success factors out of which one you're using, and that'll continue to change.

But if you're building more of an orchestrated system, you're thinking of — okay, the things that we talked about earlier: the rules and policies, past findings, things that you don't consider relevant, working with your context graph — you start working more holistically as a system. And so we have a blog post on this that I think might be helpful for you if you want to check it out.

But what goes in your security context graph? Because I sort of blew past that pretty quick.

Okay, so you think of the code paths and the trust boundaries that you have in your system. You want to think about which committers and ownership you have. It's good to actually look through your Git history and figure out what's normal inside of the system and how does this normally get worked on: who works on it, what time of day they work on it, the type of stuff they work on, what systems work together. I'll show an interesting example of something you can look for in your code. You want to look at past findings, verified repairs. You want to look at authorization flows, dependencies, routes, APIs, all that kind of stuff. So all that kind of gets fed into this graph, and that helps your adversary be a little smarter.

I'll give you one piece of research we did that has been really successful, and you should try to replicate this in your organization if you can.

We take the Git history, and we want to look for connections between services that would never show up in a call graph. Because traditional security tools would build this call graph and say, "Okay, this one talks to this one, this one talks to this one, and so we should test those conditions." But in a microservices world, especially if they're separate repos, that's not always going to happen, right?

But let's say you have a giant monorepo. There's got to be a way to tie some of these services together. Well, one way to do that is — if in your Git history, a change to your auth service and your order service happens, say, 55% of the time — there's some sort of change that's happening correlated within a time boundary window; the pull request touches those two services together — hey, maybe we should consider: can those talk to each other? And if they do, why? And why did the developer want to update those two things together? Maybe they have some sort of shared value. They're passing some sort of ID between each other.

And so now that kind of stuff can show up as in scope for your adversary to look into — and if you were just looking through the raw code, you wouldn't find that.

I talked a little bit earlier about exploit chaining. What does that really look like in the real world? We've seen this in different testing with some of the different products out there. But you could have a medium finding of, okay, the client has a writable rate limit. You have a server trust counter, and then you have two upstream APIs that are open APIs that you can access — not OpenAI, but open APIs. You have a low and two mediums, but then really it could be unbounded paid API abuse, where now you can have access to that. So you want your agent to have enough time to think about it, where it can gather those and figure out: hey, could this all be connected into one? And if you're trying to do it too fast, or you're just looking for these individual items to report on, that can be a problem.

All right. So we look across all of our system, and we said — okay, if we were to split this up into a quadrant — and I think this will be something we're going to try to work toward producing more data on — we see a lot of our leaders that are doing both patching their dependencies within 30 days or less, and getting the PR findings resolved before merge. They fit into our leaders quadrant. And then we have laggards that are maybe new in the system, or they're early in their journey, or maybe they do it in some repos but not others. For some reason, they're not making that full transition all the way.

But I think you want to see: how can we maximize both of these numbers so we're getting all our dependencies dealt with — that are found through our SCA findings or whatever — and then dealing with any findings that come in through a merge, not letting them into the code base? That's job number one. You want to stop the new risk from coming in from the AI agents.

So the idea is we want to build this paved road that agents can follow. The challenge is given to the merge candidate, it's able to figure out repair, you do a re-challenge, and then it can get accepted in.

So I've been thinking about how this has been really shifting, and I've been considering what does maturity look like now, and how do we think about this? And this is something I've been putting together for six or seven months now.

People started using prompts, and they were really saying, "Okay, that's a good way to do security." And then, okay, we'll have some skills, and okay, maybe that would work. Then we started building agents that could do reasoning. You still have some coverage problems there. And then we start with the factory — and we've been talking about that the last couple of days here — but that is where you can actually need guardrails and coordination, and your security system needs to fit inside of that, and the right proper gates put in place.

And so I started thinking about this in another way. It's like — a lot of people are going through this same journey where they're trying to make the coding visible, and then the security of the code. They want to codify it. They want to get to the operated state where they're fully assured. And they're moving from a P0 to a P3 company in their adoption of it.

So I've been putting my thoughts together in what I'm calling the AISSMM. I kind of like the name of it — the AI Software Security Maturity Model. It's like a 20- to 30-page guide that'll have some ranking questions, it'll have a grid between those four categories. I'm not really ready to release it yet — maybe next week or maybe the week after. I have a couple of edits to do, but I'd be happy to send it to you in its draft state with a watermark on it. If you send me an email to james+aissmm@dryrun.security, I'll send it to you and would love some comments, which gets me to the help I'm looking for.

So I'm trying to find stories — whether it's anonymous or, if you want to be referenced, I'd be happy to put you in the AISSMM. I'd love any comments or review of that. And then I'm also really dealing with a lot of dependency or library problems that are created by AI coding. And if you have any of those, I'd be really interested in hearing those. What do you like? What are you frustrated with, and what's going on?

So anyways, I want to say thank you for your time. Happy National Cybersecurity Month to you all, and have a good day. Bye.