Scaling Confidence Through AI-Powered Resiliency Testing

Host Intro (Gene Kim)

The next speaker is Christina Yakomin, currently working on performance and resilience optimization at Vanguard. I'm such a fan of her and her work. In 2021, she gave one of the best next-generation infrastructure operations talks I've ever seen. She's been a part of our programming committee for years, where she's helped shape this programming.

Vanguard created the world's first index fund in 1975. Now they have over $10 trillion of assets under management — I'm sure that's much bigger now. And technology has always been critical to them, evidenced by the fact that they have over 7,000 developers.

So if I can just share a little personal story, which is why I'm so excited about her presentation. It's April 2025. We were at the forum gathering that we have every year, and we were talking about how excited we were about vibe coding, how it's going to change coding. And I think Christina was at the center of a very vigorous debate of — really, not necessarily skepticism, but maybe conservatism — of, are enterprises really ready for this? Can we really use this to help support complex code bases that are at the heart of any complex enterprise?

And so early this year, she shared a story of a problem that every technology leader faces, which is, are these services that we depend upon as resilient as we need them to be? And Christina jumped in and made some incredible discoveries. And I think her experience report is going to be such an incredible contribution to the site reliability literature. So here to present that story is Christina. Thank you.

Christina Yakomin

All right. Hello, everyone. My name is Christina Yakomin, and as Gene mentioned, I'm a senior manager at Vanguard, where I work on the platforms and practices that help engineering teams prove that their systems can hold up in production. Most of my career has been spent at the intersection of site reliability engineering and platform engineering, and for the last few years, specifically on resiliency testing at enterprise scale.

And right now, we're living through one of the most significant shifts in software engineering that we've ever seen. Generative AI is making software delivery dramatically faster. This session is about how we maintain confidence in our systems while keeping our feet on the gas. I'll share the AI-powered platform vision that we're building toward, and then go deep on one concrete example of resiliency testing where the work is already making a real impact.

Whether you're calling it AI-assisted development or vibe coding, the reality is that generative AI has completely transformed the way that we write software, and pretty suddenly. As Gene mentioned, just a year and a half ago, I was still pretty underwhelmed by these tools when applied to complex distributed enterprise code bases. But that's all changed very quickly, just as Steve Yegge assured me that it would from where he sat 12 to 18 months ahead of my reality.

So now code output has increased exponentially, unlocking the potential to deliver value faster than we ever could before. But the time and effort that it takes to produce code is falling a lot faster than the cost of validating it, creating a bottleneck in the software delivery life cycle. And I don't believe that organizations can accept the traditional trade-off anymore. We can't say that we'll ship faster and accept more risk, but we also can't say that we'll slow everyone down in the name of safety. Confidence cannot become a casualty of speed, and vice versa.

Validation has to become faster and more rigorous — not one or the other. If AI is helping us build software faster, then AI also needs to help us validate software more effectively. And as more and more builders get on board with vibe coding, enterprises will need an equally intelligent way to check the vibe.

Our vision is an AI-powered builder platform that applies generative AI and intelligent automation across every step of the software delivery life cycle: requirements, build, validate, deploy, and operate.

My focus is primarily on the validate step, because the stages that surround it are already quite mature. Our CI build automation is a well-oiled machine. Code goes in, and a whole pipeline of checks runs without anyone even really thinking about it. And our CD release automation is also excellent. Once something's approved to go, it gets to production reliably and repeatedly.

But in between those two is where things get a bit murky. There's this stretch where additional manual validation may or may not be required, may or may not be happening, and nobody really has clear visibility either way. As much as we'd like all necessary testing to happen during the CI build, that's rarely how it actually happens. For example, performance testing can take 30 minutes or more — longer than every other step of the CI build combined.

Performance testing might have been happening for every change 15 years ago when we were making changes four times a year. But now, product teams are delivering software at least daily, and that practice has been left behind. Teams are performance testing periodically — maybe weekly, maybe monthly, or maybe even less than that. But there isn't much standardization around the requirement.

In the non-production integration environment, what gets validated, how thoroughly, and whether it happens at all, is driven almost entirely by human discretion and individual judgment. That's the gap that we're going after: making it clear what validation is required and when, taking the guesswork out and the manual effort out, automating as much of it as we possibly can so that you get a smooth, observable transition from the CI build to the CD release.

So does your code pass the vibe check?

What I mean by the vibe check is an intelligent quality assurance capability — something that continuously evaluates non-functional product requirements and returns actionable confidence signals. And the scope is deliberately broad. We have performance, reliability, efficiency, operability, security, and accessibility. These are the non-functional requirements that actually decide whether something is production-ready. And historically, they're the ones that get validated last, least, and the most manually.

Today's deep dive is going to focus on the top row — performance, reliability, and efficiency — because that's where we started, so it's where I have the most to show you. But before I go deep, I want to define all of these terms at a high level, because "non-functional requirements" is a broadly applied phrase that can mean different things to different people.

Performance here is about workload demand. Can the application handle the traffic volumes that we expect? What are its capacity limits, and how does response time change under load? Will this change that we're trying to introduce cause a regression in the system behavior?

Then reliability is about failure. What happens when a dependency goes down? Does functionality gracefully degrade? Can the application recover on its own? And how big is the blast radius?

Efficiency is about the cost of the outcome. Is the application using infrastructure efficiently? Are the cloud resources right-sized? Did this change quietly introduce unnecessary spend? And is auto-scaling working the way that it should?

Since these top three are where we've invested the most in AI-powered automation so far, that's where I'm going to draw my examples from for the rest of the talk. But automation of these other three pillars is where we're headed next.

Operability is about whether we can actually run the thing in production. Will our operational signals surface a problem quickly? And can a team diagnose that failure without having to get into a war room? Does our telemetry give us enough insight into system behavior? And do the alerts actually fire when we need them to? This one is particularly challenging to validate in non-production due to monitoring and alerting configuration differences between production and the integration environment. So honestly, I'm still exploring the right strategies to get this one right.

Security is about resisting malicious behavior. Can an attacker bypass our controls? Can sensitive data be exposed? And do those protections hold up under real-world conditions? Our security testing practices are already extremely rigorous, so the opportunity here is increasing the velocity of that testing through the introduction of automation.

Accessibility is about whether people with a wide range of abilities can use what we've built. Can someone complete the key task with a screen reader? And can the application be operated with a keyboard only? Will a change introduce a regression that creates a barrier for a real person?

We're still working on truly autonomous validation in these three categories, but the model is going to be the same: define the requirements, gather the evidence, and evaluate it automatically.

---

So now I'd like to share how we've applied this strategy to validating performance, reliability, and efficiency. Over the past few years at Vanguard, we've invested heavily into scaling resiliency testing across the enterprise. Roughly 20,000 resiliency tests were run with our homegrown tool suite in 2024, and that increased to roughly 50,000 in 2025. That's real growth that we worked hard to achieve, representing more teams testing more often. And by the usual measure, that's a success story.

But in 2026, we asked ourselves a harder question: of all of those tests, how many were actually telling us something useful? That question shifted our focus from volume to effectiveness — the share of tests that achieve their intended goals and produce meaningful, analyzable results.

But I want to be clear: this is not a retreat from scale. The ambition is to achieve both more tests and better tests. But if you only measure quantity, you can grow the number forever and never truly move the needle on confidence.

So in the first half of 2026, we measured the quality baseline honestly, and we found that only about half of the test runs were even valid — meaning that they successfully generated load and produced response times over a significant enough time period that we could actually analyze them. Fewer than 5% fully achieved their intended goals: hitting target throughput, sustaining it for an acceptable duration with tolerable response times and error rates. And despite how many tests were running, still fewer than 2% of production deployments had been resiliency tested at all.

I share these uncomfortable numbers because I think a lot of organizations haven't known how to get to this more meaningful data. The volume metric looked really healthy, but these confidence metrics were not. The answer to this couldn't be fewer tests. It needs to be smarter tests.

So that brings me to why we turned to AI. If effective resiliency testing is hard, expensive, and dependent on scarce expertise, then most teams will do it rarely, and when they do it, they'll do it badly. And it's not because they don't care, but because their incentives are aligned elsewhere. Their goal is to continue increasing the pace of value delivered, and unwieldy, high-effort testing just gets in the way of that.

So we've attacked the effort with three areas of AI-powered assistance. First, test generation — reducing what it takes to get a test that you can actually run. Next, intelligent analysis — making sense of what came back. And finally, contextual recommendations — telling the team where to go next.

This is a loop. Better generation produces results worth analyzing. Better analysis produces sharper recommendations, and those recommendations feed the next test with automated action to iterate until the desired outcome is achieved.

---

So let's get into the details of what the implementation looked like.

Traditionally, building a resiliency test has required an engineer to understand the application, the testing framework, the scripting language, and what expected transaction behavior looks like — which is a lot of specialized knowledge just to get to a test that you can run. Today, our automation starts from an API specification provided by the product team. AI interprets that API contract and generates an executable test script from it, leveraging a robust knowledge base that we've populated with enterprise context and testing best practices. That removes the manual scripting effort and the deep tooling expertise that used to be the price of entry. The product team provides something that they already have, and what comes back is a test that's ready to run. For now, API specification is the supported input, but additional forms of application context is something that we expect to expand to over time.

But just generating the script isn't enough to achieve high-value resiliency testing. A test can execute perfectly and still tell you almost nothing if the workload that it defines is unrealistic. If you only run a handful of virtual users against a service that handles 10,000 requests per second in production, you haven't proven much — but you've generated what appears to be a successful result.

So our solution looks at what actually happens in production. It uses existing production service data like typical throughput, previous peak loads, configured service level indicators and objectives, and typical response times. And it uses those data points to understand how the service normally behaves and how much demand it must be prepared to absorb. From that, it configures the test with the appropriate virtual user count, ramp-up time, duration, and success criteria — like the 95th percentile response time and overall error rate that you're targeting. This replaces the manually estimated configuration, or somebody's best guess, with parameters grounded in observed application behavior. So there's less research or guesswork, and a much better chance that the test actually achieves what it was intended to achieve.

Next is a big hurdle that we need to overcome, and it's one of the main reasons that we don't perform this type of testing as part of the CI build process. I'm talking here about the non-production test environment. Even a well-designed, well-configured test will give you the wrong answer if the environment that it runs against is a fraction of the size of production. You either miss the failure mode entirely, or you find one that would never actually occur at production scale — and then you send a team chasing after something that isn't real.

Our test environment is not scaled to production capacity around the clock. That would make our cloud infrastructure bill unfathomably high. So our solution temporarily scales out non-production capacity to correspond with the production provisioning. The test executes in an environment capable of producing production-representative results, and then scales back down automatically. This way we get more representative testing without asking the organization to permanently maintain production-sized non-production environments. The cost of scale is incurred only for the duration of the test, which makes representative testing affordable enough to do broadly.

It's more time-consuming to manage the test environment this way, but since automation is in the driver's seat, this is typically happening asynchronously while developers and SREs spend their valuable time working on the next big thing.

And now for the part that I think is the most important. Executing a test is not the same as achieving its purpose. A test can run cleanly end to end and still not answer the question you had when you started.

So the testing intention is provided upfront, so our system is aware of the outcome that the test is expected to demonstrate. The intention would look something like, "Reach double your previously observed peak throughput. Sustain it for 30 minutes while keeping response times and error rates within tolerance thresholds." The system will then evaluate the resulting evidence against the stated intention, drawing on other existing service and platform data sources wherever they're relevant.

If the goal was met, then we have our answer, and we have really strong supporting evidence. If it wasn't met, that's not a failure — but it's the beginning of the most interesting part of the workflow.

This is where a traditional testing tool stops. It tells you the test didn't hit its target, and then hands you a report and wishes you luck. Historically, that's the moment where a team either goes and finds an expert to dig into it, or quietly gives up.

So we built a remediation agent. When the test goal isn't met, it works to determine which layer is actually responsible for that, and then it acts. If the test script was defective, it will repair the script programmatically — wrong endpoint, bad payload, broken correlation between requests — fixes it, and reruns it. If the configuration was wrong, it will adjust the parameters. Maybe the ramp-up time was too aggressive, or the virtual user count was too low to reach the target, or the duration was too short to sustain it. It'll tune those and rerun.

But if neither of those was a problem — if the test was sound and the system genuinely couldn't hold up — then that's a real finding. In that case, it goes further and suggests specific adjustments to application code and configuration.

This is exactly what an experienced SRE does when they sit with a team: is it the test or is it the system? Telling those two apart is the judgment that we're operationalizing, and it's the thing that used to require an expert. The loop keeps running until the test intent is satisfied, or until we've confirmed a genuine finding worth a human's closer attention.

---

So now let's talk about the impact of this work. Many of the improvements that I just shared were delivered very recently, but we've already seen strong signals indicating that we are on the right track.

Since the start of July, valid resiliency test runs went from only about half to roughly three-quarters. That's a 50% increase in the share of tests that are actually generating load, sustaining it, and producing results worth analyzing. Tests achieving their stated intent are up 50% as well — hitting target throughput, sustaining it for the intended duration, and maintaining acceptable response times and error rates. That's the metric that I care the most about, because it's the one that tells us whether we're truly confident in our system's reliability.

And here's the part I really want to underline: we didn't buy that quality by testing less. Overall test volume is up 10% compared to 2025. So we have more tests, and a much higher share of them are meaningful.

These are just early indicators, so I'm not ready to take my victory lap — it's only a few months of data — but we know that we're heading in the right direction. Which raises the question of where this is going to go next.

Vibe Check is the future-state vision for a quality assessment orchestrator inside of our builder platform. It doesn't exist yet, but this is what we're working toward. Our Vibe Check will determine which product requirements should apply to a given software change, identify the evidence needed to validate those requirements, and then delegate that work to specialized agents for further testing. Those are those six pillars that I mentioned earlier. Those agents are the ones that will generate and execute the tasks, analyze the outcomes, and iterate within their own domains. Vibe Check will assess the combined evidence that comes back, delegate additional validation where requirements aren't yet satisfied, and once they are, it will confidently hand the change back to the delivery life cycle in a prod-ready state.

Resiliency testing validates performance, resiliency, reliability, and efficiency, and that's where we're making autonomous validation real today. Vibe Check is how we envision orchestrating that validation across the complete set of non-functional product requirements tomorrow.

---

To make this vision a reality, I'm going to need your help. I want to hear from others that are on a journey to automate builder platforms at enterprise scale. Where are you introducing standardization, and where are you leaning into flexibility and personalization for each team? I especially want to collaborate with others who specialize in the validation of non-functional requirements, so that we can trade notes on what to validate, when to validate it, and how to validate it across all six of the pillars that I mentioned.

Before I close, I do want to talk about people, because that's an important part of this that I haven't mentioned. Every enterprise has a handful of engineers who are genuinely excellent at quality assurance. They partner with teams. They teach people how to think about failure, how to test for it, how to prepare for what they can't predict, and they do it tirelessly. But there are only so many of them. And no matter how hard they work or how many engineers they train, we've never had enough of them to go around.

That was a real constraint. High-quality testing only happened when a team had both the right expertise and the time to apply it, and neither of those things scaled. So we stopped asking teams to do more and asked a different question instead: what if every team could have an expert SRE?

Everything I've shown you today — the generation, configuration, assessment, and remediation, everything that Vibe Check will orchestrate on top of it — all of that is capturing the patterns and the instincts and the hard-earned judgment of our best engineers and making it available to every team. Not as a tool that they have to go learn, but as a capability built into the way that they already deliver software.

And here's the part I want to be really clear about: the goal was never to replace those engineers. It was to amplify them. Their expertise is now embedded in the system itself, guiding thousands of tests across thousands of changes. That didn't make their roles smaller. It made them even more important, because now those experts spend their time on the things only they can do — defining what good looks like as our architectures change, and teaching the system how to recognize failure modes that it hasn't seen, finding the gaps that automation can't see yet, and pushing it on what it should be able to do next. Because we can scale their expertise, but that expertise still has to grow. And it's people like them who keep our systems ahead of the pace of technology.

I opened by saying that confidence cannot become a casualty of speed. That was the tension. But through intelligent automation, we've found our resolution.

We have more rigor — we're working toward every single change being fully validated against our standards of quality continuously, not periodically. And we have more speed — tests that generate, configure, scale, and assess themselves. And we have no trade-offs. We've proven this quarter that we can increase both quantity and quality of testing at the same time.

That's what scalable confidence looks like. Not more effort or slower delivery, but automation capable of better judgment, available to every team on every change. So we won't have to choose between shipping fast and shipping confidently.

Thank you.