Agentic AI and Code Reviews: Patterns and Case Studies

Host Intro (Gene Kim)

Oh, fantastic. All right. So this morning we heard from Cal from Morgan Stanley, who asked one of the defining questions that all of us are facing: when AI generates 100x more code, how do we review all of it?

So our next two speakers have spent the last six months working on creating written guidance to exactly that question. And so at this spring's forum, at the Itemized Revolution, one team took on code reviews for agentic code. Elizabeth Hendrickson, who is speaking tomorrow, led it, and Randy Shoup called it the dream team. So when they presented what they found in June, it questioned a whole bunch of assumptions I've had for decades, which includes: what are code reviews actually good for?

So to introduce Randy Shoup — I met him at Jez Humble's FlowCon conference in 2012, and he is actually the most widely cited person in the DevOps Handbook for his work that he did at Google and eBay, and now he's SVP of Engineering at CircleCI. Michelle Gill will be co-presenting with him, and she presented here in 2024 back when she was Senior Director of Engineering for Core DevOps at GitLab, where she was working on some of their highest priority AI work.

So here is Randy and Michelle.

Randy Shoup

All right. So we're going to talk about agentic AI and code reviews, and we put together a pattern language and a bunch of case studies in the paper. It's freely downloadable right now, so you can be downloading and reading it while we do our talk.

We are in this world where there's a lot of AI-assisted code. How much? A lot. So just in the last three quarters, it went from 19% of code to 57% of code — a tripling of the amount of AI-generated code in only three quarters. So that's kind of a lot.

And if you look at, as we are able to do at CircleCI — if you look at the sort of span of what companies are doing, because we have the opportunity to see hundreds of thousands of pipelines and millions of pipeline executions — we actually see a big differentiator between teams that are moving really quickly and teams that are moving slowly. And not just a little bit of difference, but a lot. So the P95 teams, the top 5% of teams, are now moving 9X faster than the median. Not than the lowest — than the median. What in the world are these 9X teams doing? We're going to talk to you about that in this talk.

The first thing that they're doing is they're removing the constraints. We all know the Theory of Constraints — every system has at least one of them, and if you focus your improvements anywhere else other than the bottleneck, they are useless, or maybe even worse than useless. So what we're trying to do is figure out how we can remove the constraints, one of which is code review.

So in our CircleCI 2026 State of Software Delivery Report, which we do every year, we see that AI-driven changes are breaking more often than they used to, they're taking teams longer to fix, and therefore validation and integration is the primary bottleneck. So now we're first going to talk about code review as a constraint, we're going to go through the patterns, and then we're going to go through some really interesting case studies that apply the patterns in real life.

So I'm going to hand off to Michelle to talk about code review as a constraint.

Michelle Gill

Okay. First of all, I hate constraints. Why don't we just remove the constraint? We are shipping all of these pull requests and it's increased nine times, so why don't we just remove all the code reviews?

I guess it turns out there is still value in code reviews, and we should keep them around. So the first value behind having a code review — of course, defect finding. It's a key reason why we have code reviews in the first place. But finding the bug should be a quality gate to production, and we're using code reviews as an all-encompassing gate to production.

So what are some of the other valuable things we would want to keep in a code review? Well, there are non-functional requirements like readability, maintainability, style guides — things that could be deterministically checked. Knowledge sharing is incredibly important, to have that tribal knowledge — that's really the key way that it builds up within your teams over time. Of course, an alternative perspective, but most importantly, an alternative solution to solving the problem. In engineering, there are so many different ways to solve problems. And then finally, kind of like I mentioned about the all-encompassing gate to production, it is there to mitigate risk so that you don't introduce something harmful into production.

So how do we shift to this AI-native development style of working, but still keep all of the value of a code review, and continue to build our confidence and trust?

So first we want to define what a code review is. There's a group of — I think there's nine or ten of us in total; you can see it on one of these slides, you'll see all of our names, and if you download the paper you'll see that too. But we really spent a lot of time in the beginning to just define exactly what code review is, so that we understood what it isn't, to start eliminating some of those things.

So I'll read the full definition: Examination of source code by an entity that is not the author, with the intention of making a judgment that leads to an action. Why are we examining the source code? Mainly for two reasons — the non-functional requirements like readability and maintainability, but the second reason is: is it intended behavior? Which is really hard sometimes for AI to catch; we'll talk about that in a minute. Why should it be someone who is not the author? That's because your head's down working, and it's hard for you to see an outside perspective until you pass it off to someone else. And then what do we mean by it leads to an action? Well, that's the difference between somebody who's reading your code and somebody who's reviewing your code. The action is going to be rejecting your changes, merging your changes, or giving you actionable improvements that you can make.

So our goal here is ultimately to remove humans from the critical path. But that's going to vary depending on your comfort, your organization, your maturity level with AI, the environment that you're in, your other constraints. So really, these three buckets are what the patterns that Randy will talk about are going to make up for you. The first is pushing as much as possible into deterministic checks and getting them out of the human cycle. Risk-tiering the degree of human involvement — I've heard that a few times today already. It's not one size fits all; you can't be super scared of AI or totally using AI all the time, there's a middle ground. And then finally, having something that is durable and inspectable every time a decision is made, an action is taken, or anything like that — and that's because that's going to get you the evidence for your trust and your confidence.

I am going to give it back to Randy for patterns. We've already seen today several case studies — especially with Dustin this morning, and then right after him, Cal. If you're wondering how to achieve the results in those talks, Randy's about to give you all of the patterns that align right back to those. And you will be surprised to learn that you mostly know them already.

Randy Shoup

Yeah. So just to take a step back for two seconds. This is obvious, but I think it needs to be said. When we were doing human authorship and human review, we could get away with less systemic rigor. Why? Because we had humans authoring it and we trusted them, and we had humans reviewing it and we separately trusted them. But if we have machine authorship and machine review, we have absolutely no choice but to systematize all the rigor we didn't have to have before. Does that resonate with people? Cool. All right. Here's some rigor.

Number one: small, incremental changes. Nothing to do with AI. This is a thing we've known in DevOps for a long time. Why? It's easier to understand, it's easier to roll out, it's easier to roll back — when done by a human, when done by an AI. We should always be separating changes that we make to the functionality — where we hold the implementation constant — versus changing the implementation — where we hold the functionality constant. This is useful for humans so that we only have to manage a little bit of cognitive load, but that's similarly true for the AI: a smaller context window is way more likely to give you the correct results you want, and it's also way more token efficient. And as I'm sure people have experienced, by default, if you give LLMs a really large thing and you don't force them to break it up, they're going to make a big old mess.

Shift left. Who's heard that before? The idea is we have an inner loop where some entity — human or agent — is doing a fast build, test, debug loop. We want to give as much real-time feedback into that loop as possible so that we can incorporate that into the commit or the merge or whatever. So obvious things that we should do here include deterministic checks like static code analysis, linting, standards checking. We're going to check those in the outer loop, but the more that we can check them and validate them internally, the less likely we are to have failures in the outer loop. And similarly, when we have specific rules about this particular repository that we're operating in, that's the kind of stuff that we should absolutely be shifting left.

Design for verification. This is my fancy way of saying some flavor of: please test as part of your development. Behavior-driven design, acceptance test-driven development — these are both extremely valuable for AI. Why? Because you're telling them upfront, in an executable way, what the intention of the code change is. So explicitly specify intent and behavior through behavior-driven design, through acceptance test-driven development, something like that. Encode your service and API expectations in contract tests or schema validations, or ideally both — making sure that the correctness of the API is machine-checkable. Focus obviously on not having your tests be flaky, because that makes the nondeterminism even worse. And then again, when we design for verification in this way, it's useful context into the agent, and it's more likely for the agent to give you good results.

Deterministic engineering practices. So we've seen a resurgence of compiled languages, of strong typing, of explicit dependencies. Why? Because they're explicit. The machines don't mind the typing. The dynamic languages, convention over configuration — I mean, it's fine, but those goals are to minimize human typing, and we don't need to do that with agents. In fact, we want the agents to have more context, more specificity, more determinism. Similarly, if you're doing something in distributed systems — like I've been doing for a long time — make sure all your operations are idempotent, again easy to reason about. And observable state transitions when you're in some kind of long-running flow, again because it's more explicit and easier to reason about. All these things are determinism.

Spec-driven development. Okay, now we're starting to talk about the AI. So we want to specify the behavior upfront. We have a structured specification. Now, what's the human doing? The human is reviewing and iterating on her intent before she sends it off to the agent to implement, and can have her agent help out with that. Ideally, how do you know if your spec is good? If all the bugs essentially are spec violations, that's a good indicator that your spec is nice and complete. And then of course you use the spec not only to tell the agent what to do up front, but you use that as a validation for correctness and compliance on the back end as well.

Context engineering. So I think we've all co-discovered this idea that we need to collect together all the implicit knowledge that the humans seem to have in their heads, and we need to give that in an explicit way to the agents. So typically now that is taking the form of some kind of knowledge graph which aggregates together documentation, systems behavior, all the things we know from code reviews and incidents, et cetera. Typically, those are aligned in-repository in some sort of metadata like `agents.md`. Again, similarly, these deterministic things like API contracts, architectural constraints, et cetera — those also are important to the context here. And then encoding security and compliance policies — this we're going to see in one of our case studies at the end.

Continuous code quality. So again, we want to make sure that that inner loop has as much context as possible and as much feedback as we can. So beyond the deterministic checks I was talking about earlier, imagine that you have a producer of code — we can call that the author agent — and you have a reviewer of code — we call that the review agent. Something we've been experimenting with in preview at CircleCI, and we're getting a lot of great results: inside the IDE if you're typing, inside pre-commit hooks, inside pre-merge hooks — and again, the engineer's goal here is to configure and tune that review agent rather than being the primary reviewer.

Adversarial review. So we've talked about an author agent and a review agent. Let's make that review agent more adversarial. So you can imagine that it's a different LLM, explicitly separating the generation of the code from the evaluation of the code — that's LLM-as-judge. And then prompting for that thing not to want it to pass, but to want it to fail — so adversarially prompting the review agent to try to find all the issues.

Multi-agent review. So one can imagine that adversarial agent is looking broadly at whether this code change is good, but obviously a better — or a more comprehensive — way to do that is to have individual agents that have their own concerns, all of which are working together. So you can imagine a correctness agent, a performance agent, a security agent, a compliance agent, et cetera. You would need some triage upfront, and you would need some ensemble coordination at the end to decide whether to accept or reject the change. But we've seen all these patterns in the case studies we've seen so far in the conference.

So now let's talk about some of the case studies we have in the paper, and I will pass it back to my lovely co-presenter here.

Michelle Gill

Hello, it's me again.

Okay. You did see case studies earlier, but we've got these three that are in the paper, and I think they do a really good job of — some of them use the same patterns in really different ways, and all three of them are really different journeys. So I'm going to walk through each of them at a high level, and then if you want to know more, go download the paper.

Case Study 1: Dryrun Security. I think James Flickett is here — shout out, James — and he's talking tomorrow. So Dryrun Security started, I think, in 2023 — shout out if I'm wrong. But they started from the beginning using AI. AI was writing the code, AI was reviewing the code. They had repository settings configured, so it wasn't going to just push blindly into production. But this is my "if your CEO comes to you and says, 'Yeah, but what if we start from nothing, from scratch?'" — this is what that would look like.

And guess what? The security review didn't work, and a huge vulnerability escaped: customer data was able to cross boundaries into other tenants. That's multi-tenant isolation, where a customer has come to you and said, "I want all of my data to only stay within my own tenant. I shouldn't be able to access other customers' data, they shouldn't be able to access my data." That's a conversation, a contract, that you have with that person. So it actually doesn't apply to everybody — it's a product-specific concern. And if you're using a generalist or generic security review, if you're publishing your code straight to an LLM for review, it might not necessarily know that you have that product-specific security contract as part of your codebase. And that's exactly what happened.

So their use of the patterns really started with context engineering — getting that product-specific domain knowledge back into the reviewer system that they were using. And ultimately, they were able to expand that into spec-driven development, in order to define: this is the intended behavior, or this is not the expected behavior. And so this way, the author and the reviewer can start their own continuous code quality loop — just like Randy was just saying — to continuously be meeting those product-specific demands.

The other thing I'll go through is multi-agent reviews, which was a really key piece of it. They did still in the end keep the general security review — that is still valuable. But now what they're also able to do, using sub-agents for example, is keep the multi-agent pattern: have a sub-agent for something specific to authentication, a sub-agent for something specific to isolation. And now with that multi-agent architecture, you can actually have a really thorough review, you can test it against your product-specific concerns, and most importantly, you can continue putting your tribal knowledge back into those specialists.

But one of the things I found really cool was their usage of adversarial LLM reviews. Actually, I wouldn't even call it adversarial LLM review in the traditional sense — it wasn't just that they were sending it to a different LLM to review the code than who authored the code. It was actually that their adversarial review was actively trying to exploit the code that was published. Is this a real solution to the security problem? Is this actually going to open a new security issue? Is this just flagging that there's a security issue when it's actually not that big of a risk? So I thought that was a really clever way to, like Randy said, try to poke holes in the solution that you're offering.

Case Study 2: Cloudflare. I really like this example because this isn't just them bolting on an add-on AI reviewer to their process. And they did try that, and it failed for two reasons — and if any of you have tried this, you know what the two reasons are. Number one, it's too noisy. And number two, not helpful. They didn't just bolt on the add-on reviewer; they actually made it an AI reviewer system. It's an orchestrated system, and they shifted it left into their actual CI pipeline.

So with Cloudflare, what can we do about the noisiness and the unhelpfulness? The first thing that they did was they broke it down into multi-agent reviews. For their multi-agents — I just talked to you with Dryrun about auth and isolation — with Cloudflare's multi-agent reviews, they've got security, docs, back-end, front-end, things like that. So that's going to help with the helpfulness, but it's still noisy. So they've got this orchestrator agent who is taking all of those outputs from the multi-agent reviews, consolidating it — not into one big block comment that they're going to throw in a human's face — but consolidating it into: "Is that an actual issue? Is this a duplicate piece of feedback? Is this even relevant?"

And the other thing about Cloudflare is they're already biased as a company towards approval — they want you to be able to submit a pull request and get it merged. So you can already merge things without resolving comments, you can merge things even though there's feedback that you should have addressed. And still, they've got all of these AI-authored pull requests that are blocking them, getting backed up, and their teams aren't reviewing them or merging them, even with that in place. And so with the multi-agent reviews and the coordinator, they can just do a full verification of the review, post the one comment that actually requires the human to act, and then merge that.

I also want to mention the system piece of it and shifting left into CI. That piece is really critical, and the reason for that is because even though it's not production, we heard a really good talk today about non-production test instances — and this is close to that. Because even though it's not production, it is still going to reproduce or simulate some of the issues that you're going to encounter in production. So for example, in the Cloudflare case study, they were able to hit rate limits, they were able to hit model errors, they were able to incorporate circuit breakers and model fallbacks. And it's because now that that review is happening in the environment that the engineers are used to working in for their own production deployments, they're better equipped to have helpful, less noisy reviews.

Case Study 3: Block. I love the Block example because this is a really good example of iterating to organizational maturity — or iterating to being more comfortable letting AI take the reins on your code review. So Block, unlike a lot of companies, were way ahead of the game with AI adoption. They had 90% of their 3,500 engineers — in just a few quarters, by the way — using AI daily. But it actually didn't matter at all because all of those pull requests sat unreviewed anyway. Engineers still weren't seeing it as valuable enough; they were still deferrable, even though all of their work was presumably getting done. And guess what the feedback was? Noisy and unhelpful.

So what did they do? First thing they did is took a step back, sat with a pilot team, and made that AI reviewer really good. They made it specific, they gave it tribal knowledge, and when they were done with that small team, that AI reviewer really was like a first verification pass. So then the next thing they did is they shifted left locally — before you even submit the commit or push to the cloud, that same thing has already happened locally. So you've basically already got two reviews when you push something up: the review that happened locally and the review in CI.

They can also now build on that to do a continuous code quality loop. If you're going to review my code locally and you've already written it locally, you might as well fix it locally. And the same thing once I push it up. So now instead of two reviews, you potentially have two rounds of fixes that have gone in. They did break that down into multi-agent reviews; they broke it down into adversarial reviews, until ultimately they were able to put that into CI with the same concept — doing the continuous code quality loop with the author and reviewer. Maybe their database agent has given feedback, and maybe their security agent has given feedback, and the author is fixing it, and they've got another review.

Ultimately, the results of this left them only needing to provide the product-specific context engineering knowledge. And with that information, now it's a workflow — and non-developers can file a bug and click the button, and this whole chain has gone off.

Randy Shoup

Cool. Just to finish up: we talked about code review as a constraint and how we could release it. We talked about a bunch of different patterns of review that you've seen illustrated in our case studies and in the case studies from other people here. We talked about some of the case studies that you could read more about in the paper.

I just want to leave you with: it is absolutely impossible to produce code at machine speed and continue to evaluate code at human speed, and we hope that these patterns are helpful to figure out how to do that.

Thank you very much. The help that we need is: what are additional patterns we missed? What are some additional evolving practices that we didn't have when we started thinking about this six months ago? I'd love counter-examples — "we tried this pattern, it didn't work, it was a trade-off that wasn't worth it." And then any additional case studies you have.

Thank you very much, and enjoy the rest of the conference.