Las Vegas 2019

Lightning Talk: Using Your CI/CD Pipelines to Create Better Governance

Lightning Talk

SN

Scott Nasello

Director, Delivery Engineering, Columbia Sportswear

Transcript

00:00:02

So before I start John smart disasters me is your talk funny? And I said, no, I have 19 slides on vegetables, but we'll go ahead and start. So our delivery teams at Columbia are, have always been pretty autonomous. And so when we designed our pipelines, we wanted to honor that autonomy create the experience of full ownership, as well as give them the opportunity to customize for their own needs

00:00:36

And in the ideal case, I don't think my slides are moving and the ideal case, um, the teams there, again, in the ideal case, uh, the teams are stewards of that responsibility and freedom and they actually do the right thing. But the reality is that sometimes those teams will optimize for their own incentives or maybe even, uh, for their own delivery. And that certainly isn't the enterprise, uh, best, ideal.

00:01:05

Okay. We've been heavily inspired and influenced by a Topo pal and capital one. Uh, if you haven't checked out the software, uh, clean room, it is amazing. It has a lot of focus and rigor on attestations, uh, artifacts and, uh, and really limiting access to production, which I think is pretty awesome. But we've taken that, uh, point of view and we've actually engineered it into our pipelines so that we're enforcing those, uh, foundational gates that will, we get to better outcomes, less variation, and ultimately more reliability in our pipelines. And as an aside, it helps us to, uh, to troubleshoot those pipelines because they have a lot of, um, uh, commonality in them. And so pipeline templates and, uh, containerize pipeline services have been a huge unlock for us. It helps us to drive a water repeatability. And the fact is we have hundreds of function apps that are all using a centrally managed controlled version, controlled, uh, uh, template, which gives us a lot of repeatability. And the fact is we're trying to make doing the right things easier than anything else. And so that really starts with better defaults, whether it's encryption, whether it's application settings, uh, service, bus integrations, et cetera. We want to get to this point, which is kind of Nirvana, but we want to get to automatic compliance or continuous compliance that John Rez was talking about yesterday.

00:02:33

I think we missed a slide. I'll keep going. Uh, okay. So we want to reward teams for doing the right things. And so, uh, fundamentally that you, you know, you have these teams in your environments, the teams that can make high confidence changes at speed without impacting other teams. We want to reward that and, uh, and be a lot more on the carrot side than on the stick side, but with guard rails, our episodic and change management teams, they've been wonderful partners on this journey and they've been experimenting, developing, and testing new ways to have a lighter weight, um, governance into the pipeline. And I'm going to share four examples because everything is templated. Uh, our emphasis team can all of a sudden now own services that matter to them, things like credential scanning, and that gives them a lot of, uh, uh, autonomy and a drive to actually own those processes.

00:03:30

Another example are code scans for vulnerability or bugs. And so this not only has the effect of giving feedback to this teams that need to improve their practices, but it really elevates an InfoSec team to be first-class citizens within the pipeline. And they actually have within the vise stream and they actually have a control plane to, to drive the things that are important to their objectives and chaos squirrel our delivery, or excuse me, our developer tools team wrote this to drive better governance within our cloud environments. We can run it in either a what F mode or in a destructive mode. And, uh, it's driving a lot of great things. I have stickers by the way.

00:04:09

I was lucky enough to work on a team, uh, in the spring about 15 change management and fixing the calves specifically, we came up with a lot of ideas, it's free the document, so you should download it. And we talked about having a chain service that really exposes, um, a chain service that could run in a pipeline. And so, uh, in our example, we've optimized that chain service for standard change. And so that really means that teams are preauthorized to, to run at their own speed because they've demonstrated the rigor and that's really continuous deployment. Uh, at the best case, it can run in either a build or release, uh, context. And, um, as an example, our platform engineering team, they typically use a lot of, uh, pull requests and merge requests for, um, self-service and those run on builds. And we want to capture that into our change management system.

00:05:03

And so within the CMDB, you'll see a CIS either at the team repo or, or pipeline, and they all have different reasons for being there. But the key thing is if you apply the SRE error budgets, you can apply it to this domain. And this is really, uh, change error, budgets and teams that actually start to degrade. You can degrade their pre-authorization. So if you have a pre-authorized change that goes automatically within the pipeline, you get your corresponding record and your change management system wife is good. On the other hand, if you're a, if your chain surface area is rather large, or if your team that's developing, that would be categorized as a normal change, we're going to block that deployment to production. And we're going to invite you to have a conversation with people that care about the outcome. So after approval, whether it's a B cab or virtual cab, maybe a show of hands or ad hoc cab, uh, and you, uh, secure your, you can re trigger that pipeline to go through. And so, regardless of whether we're talking about a standard change or normal change, we're linking to artifact that was actually produced in the pipeline to the change record. So you're getting the attestation, you're getting to the audit trail, if you will, that make auditors and stakeholders happy. I think pipelines are a team sport, and this is an opportunity to bring your governance folks into the conversation to help them to, um, to drive behaviors and, uh, start to own some of those artifacts. So thank you for having me.