Europe 2022
VendorDome: Can You Accelerate Delivery Without Compromising Security?
This session is presented by CloudBees and Sonatype.
SM
Stephen Magill
Vice President, Product Innovation, Sonatype
HL
Hope Lynch
Sr. Director, Technology Strategy, CloudBees
LD
Logan Donley
Sr. Technical Evangelist, CloudBees
Transcript
Welcome to today's vendor dome, where we have Steven McGill, Logan Donley, and our topic is can you accelerate delivery without compromising security? Everybody wants to go faster, but what does going faster? Well look like. So, uh, first let's get our introductions done. Um, Logan is, is here with me from cloud B's Logan. Tell us a little bit about yourself.
Yeah, so I started here at cloud B's as a solution architect and have to apologize. I'm a bit sick right now. So my voice is probably not super strong, but hopefully you can hear me. Yes. Um, but now I'm a technical evangelist and work with, with hope and we, you know, work on kind of helping our customers understand really what we can do with, uh, you know, modern DevOps practices. Mm-hmm
<affirmative> all right. Thank you. And Steven, Steven's here with us from so type.
Yeah. Hi. So I've been working in software security ever since, um, doing my PhD in that area 15 plus years ago. I won't say how big the plus is. Um, but, uh, yes, you know, sort of initial my initial work was in sta static analysis and, and reasoning about programs. Um, since then I've gotten sort of more and more interested in, um, development workflows and the processes that you can put into place at large organizations to actually, um, sort of apply tools in the right way, have, have a big impact, uh, in terms of software security. And so, um, that's been my focus, um, here at Sonotype. Um, I've been, uh, lead for the last three software state of the software supply chain reports, which is the annual report that Sonotype puts out. Um, and so we learn a lot each year with reporting about, um, how the community is managing open source, how enterprises are managing their use of open source. Um, so I'm really excited today to, to talk about this topic and you know, what, what people out there can do, uh, to really substantially improve their security.
Excellent. And, uh, and I'm your moderator Hopely? I also work at cloud B senior director platform. My background, uh, I've been a product manager for IOT platforms. I've managed systems engineering teams when they were called systems engineering teams. I've managed platform teams, done web development, done, um, corporate strategy and, uh, one of my passionate things that I've done and still continue to sneak in where I can anything related to transformation, uh, digital, agile business transformation. So, uh, also a reminder, you guys already know through your questions into speaker track four. Um, we want this to be, uh, a back and forth conversation if we can. Um, so the first, first thing we want to, uh, establish here because, um, this rapid innovation, uh, the way we're discussing it today is tied to progressive delivery. So Logan, uh, if you can, can you tell us a little bit, um, what is a good definition or description of progressive delivery?
Yeah, that's a great place to start. So progressive delivery is moving away from having like a, a large release where we roll out just one big bundle with, let's say 10 features out to, you know, our production environment. Instead, we look at it from a feature perspective where I can roll out each feature individually to a small cohort of users. So we can start with beta users or we could do a random 10%. And the reason this is important is because, you know, if we think about a situation where, you know, Steven hope and myself have all written a new feature each and unbeknownst us the time my code is actually bad. Mm-hmm <affirmative> once it's out in production, it's gonna end up causing an outage. So now we missed it in testing. And now if we rolled it out completely, that means that a hundred percent of our users can't access the application, a total outage mm-hmm <affirmative>.
If instead we used a progressive approach where we started by rolling at say, 10% of our users. Mm-hmm <affirmative> well, it sucks for the 10% who can't access it, but the other 90% can keep using it. And you know, it's not a total failure, that's a good step, but the problem is now, you know, Steven and hope your functions, your features are kind of beholden to mine until I figure out why mine's busted, right? And so if we do something with feature flagging, this is where progressive delivery comes into play. Mm-hmm <affirmative>, each of our features can be wrapped behind a feature flag. And what that allows us to do is, and this is effectively an if statement, if you're not familiar with feature flags. Yeah. You wrap your new features behind an if statement and the conditional that determines that is pushed outside of the code.
And so it can be, uh, updated on the fly. And so you can determine based on who the user is. Uh, so we can do things like if they're a beta user, they'll see this, or if it's a random percentage, they'll see it mm-hmm <affirmative>. But the beautiful thing is we can just shut it off. And so if my thing breaks, mm-hmm, <affirmative> rolled out to some percentage of the users. We see some errors, we turn it off and then instantly, you know, Stephen and hope your features can be used by customers. I can go figure out why, you know, my thing's wrong and nobody's anchored me for a break light data.
Yeah, exactly, exactly. I think, uh, progressive delivery. I know at least, uh, in my last organization, uh, working on platform, it, it was a fantastic thing to be able to, to utilize that because it was such a, a, a critical service, uh, for our end users. And if something did go wrong, like you're saying, um, we could isolate it, maybe roll it back just for a specific set, if they had some configuration that was just in conflict. So, um, and one of the other things, though, this leads to is progressive delivery, perhaps being one of those things that does speed up innovation, right. Um, because you know, I'm always gonna roll everything back to agile. You know, if you've known me long enough, we're gonna talk about agile at some point. Um, but that iterative, you know, uh, process that you go through, you learn, you, you figure out, you know, what are the changes we need to make? How do we adjust? And, and then you try and implement quickly. So I think progressive delivery, um, for, for agile is, is just such a, a good match and does help that innovation just, just speed up so much.
I, I have a question about this. So mm-hmm, <affirmative>, um, you know, I, I, what you were talking about Logan was, you know, the ability to, um, sort of, I guess, have more reliable releases quicker rollbacks, you know, sort of preserve functionality in the case where, where there's some breakage, um, do you also see progressive delivery, uh, improving, just delivery, time, delivery, frequency, you know, uh, shortened delivery times, uh, things of that nature, just, you know, even when things work, does it just help things flow faster?
Yeah. Yeah. That's a great question. So I think it really depends on the organization. Like that is definitely the idea is yes, it should speed that up. It does, you know, there's a bit of a cultural, uh, component to it. You know, if you're going from doing monthly releases to now being able to push anything, like you just need to have your team build up the confidence that yes, we can push things faster without as much testing. Um, though I'll say you always wanna be cognizant of the fact that, like, you don't want your users to be Guinea pigs. You don't want them. Yeah. I'm sure we've all experienced it with an app that like, every time you log in, everything's shifted around mm-hmm, you're, you know, it's, <laugh> getting modified all the time. Like, you don't want that experience, so you wanna make sure things are properly tested and, and all that beforehand mm-hmm <affirmative>, but absolutely once you kind of get the organization behind that whole way of working, you can start like Steven, if you write on the feature that like, maybe you've written some code that I wanna reuse, but for whatever reason, it's blocked up, we can't push it to production yet, but we have an opportunity to just put it behind a feature flag, not enable it in production, but we're able to keep it off of along the feature branch mm-hmm <affirmative>.
And so we can kind of speed up that velocity of, of just the, the standard like deployment, uh, cycle. Yeah.
And then sort of, there's less delays waiting, waiting on large merges and, and blocking behind other people's changes. Yeah,
Exactly. Well, I should say, you know, I think it's very dependent on your organization. Like I know Sonotype as well as cloud BS, a lot of our stuff is heavily self-managed focused. And so of course, that doesn't really work in that world where like, I can't release a new feature, um, and just send it out to 5% of the users, ask them, Hey, try this out, let me know if it fails. Yeah. I'm gonna lose this 5% of users pretty quick. So it doesn't work in that world quite as much, but for, you know, the SAS world, like our, our future management solution itself is, is a SAS mm-hmm <affirmative>. And so we, we heavily use that there, cause it just, you know, we have that flexibility.
Yeah. Yeah. And, and we are seeing sort of more of a shift to, to SAS. You know, obviously it depends on sort of the industry segment and so forth, but um're sort of more comfort in, I think using cloud services and, and I think a big part of it is, is what you just said. Right. You can get access to, you know, <laugh>, if it's well managed, you get access to features, you know, in a timely way. And, and it just sort of improves things without you having to do, um, manual deploys and so forth. Um, mm-hmm, <affirmative>, you know, hopefully you're not in that sit scenario, it described where everything, everything in the UI changes on you and then flips back the next day. And then you sort of
Mm-hmm <affirmative>
Have whiplash.
Yeah. Uh, this next, this might be a moot point because the descriptions and the conversation has been so good, but if someone is new to this still, and they say, you know what, we're doing continuous delivery. Why do we need progressive delivery? Right. Uh, we're we're sending things out fast. We're we're, you know, we can, we can make changes, we can make updates. Um, how do progressive delivery and continuous delivery work together and, and how are they different for, for developers?
Yeah. So I think it's important to just consider that progressive delivery is I've heard it described as like, uh, continuous delivery plus plus like it's, it's adding on, like you still need cuz progressive delivery is really at the end of the day about introducing feature flags mm-hmm <affirmative> and that lives in your source code. And so you're introducing that at the start of the cycle when you're writing your code and then you add it to the end, after the thing is in production, you then can toggle the flags on and off mm-hmm <affirmative>. And so really you still need the same delivery mechanism to actually get, you know, your, your code deployed into the production environment. So you still need the whole CD process in place. It's just, we're adding on, you know, both on the authoring of the code side as well as at the end, then toggling on those flags.
Mm-hmm <affirmative> mm-hmm <affirmative> excellent. Thank
You. Yeah. And I, one other thing I wanted to point out is, uh, you know, sort of, we mentioned the, the advantages in, in terms of deployment frequency and speed to speed of development, um, workflow, uh, that progressive delivery has, you know, I think that ends up being really important from a security perspective as well. You know, we talk a lot about like how quickly can you remediate issues? Uh, how quickly can you respond to incidents? And a lot of that has to do with, um, sort of the capacity that you have to push changes quickly, right? Yeah. Um, whenever you have to remediate something and respond quickly to an incident, um, you know, if you have sort of that muscle built up right where you can you're you're on a regular basis, pushing things quickly, it becomes sort of, not an exception, uh, to what you normally do, but you know, just part of standard day to day mm-hmm
<affirmative> yeah. We have a question from Chris, how do you deal with this when you have legacy code that maybe doesn't have nicely decoupled components?
Ooh, that's an interesting question. Yeah. Yeah. So I mean, legacy, uh, legacy code, um, it is gonna depend heavily on, on what that particular stack is, but the thing is you, you really don't need to go all in, on feature flagging all at once. I think thing is you can kind of intersperse it where possible, like there are many components where it's just never gonna make sense. Think about a database migration. You can't really just toggle that on and off mm-hmm <affirmative>, but I think that's very true of, of larger applications where it's just gonna be too complex, um, to kind of break it down, such that any new feature can just be toggled on and off. So I think you need to start where it makes sense and you know, maybe this breaks down over time. Maybe it helps with that. But I think, you know, one of the challenges that a lot of people, when they adopt these practices make, is run into is just trying to take on too much at once and trying to deal with, um, you know, what, what all <laugh> like, it's such a huge ask. You've got hundreds of thousands of lines of code to then figure out how to kind of deal with that all in one, you know, fill SWOP
Mm-hmm <affirmative>, um, follow up on the subject subject of having quick access to remediation and rollback. And this comes from Quinn Daley. When is the right time to remove a feature flag and all the associated old code from your code base? And how do you ensure that old code is removed along with the flag?
Yeah, so that's, that's always a fun one. Um, so, you know, we cloud use actually try to make that as automated as possible by, um, kind of detecting when, when that sta point is. But really what you wanna do is once you've gotten the point where a flag has been in action, whether it's it's turned on or off for everybody consistently for, you know, X amount of time. Um, and there hasn't been any variation mm-hmm <affirmative> then it's usually safe to remove it. Mm-hmm <affirmative> if, if we've decided that everybody for a week or a month, or, you know, it is gonna depend on how comfortable you feel with making those changes. But if we can say that it's, you know, everybody has seen this new feature as true for a week, then we can probably safely start to remove that mm-hmm <affirmative>, that's where life cycle management tools come into play to figure out, okay, I've seen that this is, uh, you know, it's, it's been an action. It's what we call stale. Um, here's all the references to it. And go ahead and remove those mm-hmm <affirmative>, you know, it's just important that yeah, you don't just clutter your code with thousands of, of, you know, flags that are no longer niche mm-hmm
<affirmative> okay. And we have a question from gene, um, he says, it's amazing how future flags have taken off in the industry. What else can we learn from dynamic runtime environments where we enable the runtime to be truly dynamic? What else could we enable versus things we view as static right now?
I'm need to read that again.
I know, right. I can do it again. I can go for it. Um, basically, um, so if we have a dynamic runtime environment and, and things are happening, uh, you know, just in this dynamic di super dynamic way, uh, is there anything else that's happening in a static fashion that could be more dynamic? Um, like, like feature flagging that maybe right now is reliant on, you know, uh, uh, a manual code push or, or something.
So one, one thing that comes to mind for me is, you know, managed services, right? Mm-hmm <affirmative> so, um, again, it's becoming more common to, you know, deploy and, and not sort of manage your own database, but rather use a managed database service, use managed, you know, message, queues services, things like that, you know, um, that basically takes a component and, uh, says I'm gonna shift from, you know, having a static view of this and knowing exactly what version is it is and everything that's going into it and sort of manually updating it myself and saying, you know, it's gonna become this dynamic component that offers this API offers. This service has, we have a contract, you know, in terms of what interaction with that service means, mm-hmm <affirmative>, um, and that's sort of all that's required and the more you can shift away, you know, I think the more you can shift away from, you know, specific code into contracts of interaction, mm-hmm <affirmative> between components, you know, sort of that's the, that's the key thing that, that keeps the level of abstraction sort of correct for the problem you're trying to solve.
Mm-hmm <affirmative> mm-hmm <affirmative> okay. Um, excellent. So Philip Day has a comment on, on that question, uh, that gene asked like available compute resource for a process. So, um, definitely I think that's something that could fall into that category. Uh, so now rolling back a little bit closer to, uh, the topic of, of our conversation. Um, so with all of these things happening so quickly, so, you know, there's so much more speed, that's coming to so many more teams, um, how secure can we be with, uh, progressive delivery and automation? Is, is it inherently less secure for any reason?
That's always been my biggest question and, you know, I'm, I'm not a security expert, so I'm really curious to hear what Steven has to say, because my assumption would always be, if we're moving faster, there's, you know, we're giving up something for that. Um, mm-hmm, <affirmative> increased velocity does seem inherently risky. Mm-hmm <affirmative>, but we're the same CD processes. So maybe it's not definitely like to hear what you have to say about that.
Yeah. Yeah. I think that's, you know, a common view that like, yeah, there's a trade off between, um, efficiency speed and, and, and between that insecurity mm-hmm <affirmative> um, what we found actually. So in, uh, last years, uh, or no, two years ago in the software supply chain report, we, um, we did a survey of, uh, development practices and outcomes at, at large enterprises and found that, um, you know, there were sort of four clusters, um, that we saw at a couple of correspond to what you said, you know, it's the people moving really fast, not caring about security. You know, those certainly do go hand in hand in certain, uh, contexts. Um, there's the people who achieve security by slowing down the process, introducing checks, you know, along the way. Um, and that is an effective way to get security, but obviously slows things down.
Uh, but then there was this cluster that was really interesting, that was, uh, managing to achieve both high security and high development velocity. Um, and you know, what we saw in that group was a lot of leveraging of tooling. Um, a lot of, uh, sort of adoption of C I C D practices, agile practices, um, and then a lot of, uh, sort of leveraging of policy and centralization and sort of uniformity of their build pipeline. So, you know, if you, if you have sort of a standard for, this is what a pipeline looks like at our organization, this is how we're gonna leverage tools to do those checks instead of manual processes to do those checks. Um, then you really can raise the bar in terms of, um, uh, security while, while still having a very agile and very, uh, efficient development process.
Mm-hmm <affirmative> excellent. Um, so I've, I've posted a question to the audience. If you guys haven't seen it and you have some input, what are your best practices for day to day dependency management? Um, I think that's definitely gonna be gonna be of interest to us here, uh, on the panel having this discussion when we're talking about day to day management of, uh, security risks. Um, so do you have any, any, uh, best, any additional best practices Steven that you think, um, our audience should be aware of?
My, my biggest best practice for day to day dependency management is, uh, just do it <laugh>. So there's a, a lot of places where, where that's not, you know, something that's thought about, right. It's not part of, uh, you know, maintenance or, you know, it's not something that's, um, that's planned for every sprint, you know, mm-hmm, <affirmative>, let's revisit dependencies. Let's see if there's updates to do there. Yes. Um, but it really is, uh, super important, uh, in terms of decreasing security risk, you know, um, a lot of, I I'd say most disclosures of vulnerabilities happen by, you know, someone in the white hat, security research community discovers something works in private with the project that has that vulnerability to get it remediated, get a new version pushed out, and then after the community has had time to adopt that updated version, then it's disclosed. Right. And so if you're staying up to date, if you're adopting those new versions, by the time that vulnerability is disclosed, you're already covered. Right. Yeah. Um, and that, and so yeah, that day to day practice, uh, is really important. Mm-hmm <affirmative> so just, yeah, just, just doing it is important.
Excellent. Excellent. Thank you. Um, so Les we're, we're shifting now, uh, we're gonna shift more into your world now, Steven, a lot more. So, um, how, how can organizations, um, when they're looking at security, how do they know what they actually have? And if they have a large enterprise, right. And they need to take on remediation, um, how, how can they get their hands around that?
Yeah, it can be really hard, um, at, at large scale, you know, when you have thousands of repositories, um, to, to even know what you have. And, you know, I think the, the first step is, um, having tooling in place to, uh, generate software bills of materials for everything that you have. So S bombs you'll hear that term a lot. Um, and that really gives you sort of the start of that inventory. Mm-hmm, <affirmative>, um, there's additional information that you'd like to have on top of that, you know, not all say you have 2000 repositories, right? Mm-hmm, <affirmative> not, all of those are active applications, not, not all of those even became sort of production applications. Right. A lot of them are sort of small projects or proof of concepts. Mm-hmm <affirmative> um, so knowing which is which, um, among the applications, knowing which ones are in production, which ones are, um, you know, out there, internet facing outside the firewall, you know, those all start to give you, um, a really good sense of where does your security risk lie, right.
Mm-hmm <affirmative>. And so then when, uh, you do have an incident, uh, something like log for J which didn't follow what I described previously, the typical sort of responsible disclosure timeframe, right. Log for J the exploit was posted on GitHub, sort of to the surprise of the world. Mm-hmm <affirmative> when something like that happens, uh, then you have the information you need to go and remediate that and further more to prioritize remediation. Right. Cuz you, you know, mm-hmm, <affirmative>, you know, these are our, uh, apps that are most at risk based on, you know, sort of their network connectivity, the role that they play, you know, touching customer data, things like that. Mm-hmm <affirmative> um, and so you can just sort of then, then focus on the actual remediation rather than getting stuck in just trying to figure out what you even have.
Mm-hmm <affirmative>
So related to that, I'm just curious, you know, obviously developers are involved in your experiences. Is it typically the developers that are keeping track of these things or do most companies have, you know, someone like shared services or security team, someone that's actively monitoring the stuff and then like assigning those tickets?
Yeah. I think this has to live at a level above individual development teams. I mean, development know a lot about their code base and their application in which portions of it are security critical and so forth. And so they certainly should have a role in this inventory process. Um, but, uh, it's really an organization wide concern. And so you want, uh, you know, some, some amount of centralization of that, that inventory. That definitely makes
Sense. Mm-hmm <affirmative> for sure. So I'm gonna post another question to the audience and, uh, let's see if there's any input here. So, uh, what differences have you seen in how various tools are helping you remediate or address, uh, the security risks that you've seen, uh, in your organizations? Because I tell you from the time I started, um, in technology until now, I mean, it feels like it, it was, uh, things would crawl, you know, there would be, uh, something and it was the reverse of what you said, Steven, I would read about it before it was, you know, it impacted me, uh, often because it just moves so slowly where now, uh, it's, it's a global threat before you even, uh, wake up in the morning. So understanding how those, you know, the, the tooling can be impactful, but, but I think what you also said, um, just getting people mobilized to do it is important.
Mm-hmm <affirmative> um, one, one of the other thoughts and I read something about this, um, it was from Gartner actually yesterday and it was talking about how so many organizations will have a top down mandate be secure, be secure, be secure mm-hmm <affirmative>, but it's really not that effective if you don't have the processes in place and you have not planned it, uh, into the organization. So in your experience, um, how, how do you, how do you get even leadership to shift and support, I guess the tooling and the implementation that's that's required in, in the conversations you've had?
Yeah. I mean, it, you're totally right that, um, you know, a top down mandate on its own does not do anything, right. If you say we have to be secure, we have to prioritize security, but then you don't, uh, give developers time each sprint to address security issues. You don't have sort of a process in place to, uh, prioritize security issues to do architectural reviews and weigh in on security, you know, so that the security team can have a voice. Um, you know, even in application design, uh, where relevant, uh, then, you know, it's, it's not gonna do anything. And so, um, you, you know, from a leadership and you know, those, those sorts of structural, uh, aspects of supporting security do have to come from leadership, right? You need to say, uh, yes, we wanna focus on security and these are the processes we're gonna put in place. Uh, this is the support that we're gonna give you to make sure that we, that we accomplish that
And support sometimes meaning the dollars often the dollars. I mean, you, you, you know, doing it with, uh, chewing gum string and tape and, and, and having people build in things manually, uh, is, is just not
Fascinating. And it's not just, it's not just money for tooling either, you know, it's, it's the people, right? There's a lot of organizations out there that have like a one to 100, uh, ratio of security professionals to developers. Right. And it's
Hard to get good security people. Yeah. So, yeah. Yeah. Um, so security is not one size fits all. There are differing, um, categories of risk, levels of risk impacts, um, depending on what the risk is. So what are the different types of solutions that organizations, um, and even security professionals probably need to look at, um, depending on those, those response types that are needed.
Mm-hmm <affirmative> yeah. There's, um, you know, I I'd say yes, the, the needs are very different at a very large enterprise versus, you know, a smaller team mm-hmm <affirmative>, um, you know, the, there are, what's great. Is there are, you know, some really good, uh, lightweight, accessible tools out there for, you know, small and medium sized teams. Um, you know, there's, there's a lot more, uh, open source security tooling than there used to be. Mm-hmm <affirmative>, um, you know, at Sonotype we have, um, the Sonotype lift product, which is a SAS product that integrates with GitHub. So if you're on GitHub, you know, you're a small team, you're, you're starting to build your software, uh, or even, you know, a medium and largest team. You can, you can go use services like that, um, that are a managed service. You don't have to install anything.
You don't have to, you know, manage anything or worry about comput costs. Um, and you can get a lot of insight into the security of your code, uh, that way. Um, and then, you know, I think where it shifts is when you get into that world that I was talking about before, where you have just such a massive inventory of applications and, and code, you know, even non application code, um, that management becomes sort of the primary bottleneck, right? It's not, um, it's not monitoring this code base for security issues. Mm-hmm, <affirmative>, it's just figuring out, you know, where is all the code, making sure it's all covered, making sure your processes are uniform. You know, when it's a small team, you can agree on a process and follow it done, right? Mm-hmm <affirmative>, you have, you know, tens or hundreds of development teams, you know, it's a lot more effort to get uniformity, um, and, and full coverage across all of your code.
And so there, enterprise tooling comes into play, you know, at Soo type, we have the life cycle, uh, uh, portfolio of products. And, and so that gives you sort of that enterprise level visibility, um, where you can, you know, get that high level view of, you know, these are all my applications, this is where the risk is. And, you know, I think the other thing that comes into play at that level as well is, are things like, um, you know, setting up something, something at the gate, right? So we often see large enterprises, um, set up a repository server, something like nexus repo, where, you know, that's sort of, proxying, uh, access to open to outside open source. And that's a great place to inject additional policy to say, you know, we're just gonna check things as they're pulled in to the development pipelines, uh, to make sure that they adhere to our policy to, you know, catch, uh, zero days and, and sort of security issues, uh, sort of as they happen, just gives you another point of control. You know, it's all about that scale. It's all about having multiple points of control so that you can get that full coverage, because it's hard to get that with sort of just one point of integration.
Mm-hmm <affirmative> do you overall, do you, do you feel that, um, the state of security and organizations is actually improving a lot or is it, is it pretty, pretty much at the same level it's been same level at Smith?
No, I think so. You know, I think there's been substantial improvement. Um, I think, you know, there's, there's still a long way to go. Um, you know, there's, <laugh> like, you know, we were talking about day to day, um, dependency management, uh, in, in last year's software supply chain report, we found that like by and large people were doing a good job managing the dependencies as they're updating them, you know, choosing good versions, choosing secure versions. Um, but they were only actively managing 25% of their dependencies. The other 75% sort of never saw an update over the year timeframe that we were looking at. And so, you know, there's a lot of room there I think for, for improvement in practices. Um, but you know, at the same time we are seeing, uh, you know, the security process work, the, the timeframes for remediation come down, mm-hmm <affirmative>, um, you know, we've seen, uh, in the open source community, the average time that it takes, uh, a package to update its dependencies when, when new versions come out, mm-hmm <affirmative>, uh, has, has dropped, you know, so that helps a lot with, uh, this transitive dependency problem of like, I have, I have a package I'm using and one of its dependencies is vulnerable.
So I have to wait for it to update now to sort of patch myself, you know, so those flow down more quickly. Um, and then when long for J happened, we saw, you know, within, uh, within two days more than 50% of the downloads from Maven central were the patched version. Right. Mm-hmm <affirmative> so there was sort of very quick update of that patch. Um, it then sort of leveled off. And so you have sort of a cohort that, that is not getting the message <laugh>, you know, mm-hmm <affirmative> but, um, but you know, I think that that fast mover cohort was probably smaller, you know, 10 years ago. Mm-hmm <affirmative> so, you know, I think there has been improvement, uh, yeah, but there's still room for
More, uh, we have one from gene, uh, he says, I'd like to hear about stories of how engineers have successfully rationalized their dependencies, what things should inform, what components should, and shouldn't be used.
That's a great question. Um, I, I let's see, I, I don't know if I have any like specific stories. I <laugh>, I think it is common. You know, one, one thing that, that we have seen is, you know, cases where there's like, you know, 15 different XML, parsing libraries, you know, in use as an organization. And like, I didn't even know there were 15 X mile bursting libraries, you know, it's like, um, you, you can definitely get into these situations where pretty much everything that can be used is being used. Right. Mm-hmm <affirmative>, and that, and it's easy to get there because, you know, every, everyone has their toolkit that they're familiar with, you know, mm-hmm, <affirmative> new developers join all the time and they bring their favorite packages. And so, um, you know, this, this sort of dependency bloat thing is, is real mm-hmm <affirmative>.
And, um, and that's where, you know, having some sort of process for saying, you know, look, this is our tech stack. We're gonna, you know, we'll make exceptions, you know, and reasonable, but we're gonna try and stick to this, uh, collection of packages that helps a lot because then, um, you know, it's less to monitor for your security team. The, and the more familiar you get with, you know, the packages you're using the, the more on top of, um, security and update, uh, you can be, um, I don't know, Logan, if you have any specific stories that come to mind.
Yeah. I, I can't really think of any, but I, that is definitely always a balancing act of like, we wanna stay productive, but mm-hmm, <affirmative> like, and, and being productive might mean bringing more dependencies that you're familiar with, but that can obviously come with the security risk, you know, you're adding additional dependencies. So yeah, that's always, always a challenging point to kind of, uh, tow that line. Mm-hmm
<affirmative> um, we have a question from Billy Hudson, how, and where are people recording metrics for resolving dependency issues from discovery to resolution?
Yeah. Um, so, I mean, we, we sort of call that timeframe, uh, Mt. To you, or mean time to update mm-hmm <affirmative> um, when it is just an update, you know, not necessarily security relevant, if there's a security, um, you know, like a CVE against the version that you're moving from, mm-hmm <affirmative>, uh, we would call that or re mediation and look at meantime to remediate, um, this dependency issue. Um, and so, uh, yeah, we, we sort of measure that as, um, the time between when a new version of a dependency becomes available, uh, and when a new, a version of an application that uses that dependency gets pushed, that that has updated that dependency. So sort of that timeframe between availability and use, um, you know, in your application mm-hmm <affirmative>. And if you look at that, you know, across all the dependencies, um, that you, that you incorporate, uh, then you can get a sense for how quickly you're reacting.
Um, and, you know, if you have a regular process for reviewing dependencies, you know, that'll naturally work to lower that number. Um, and you know, the other thing about that number is, uh, you know, it's, it's not just a fun metric to track. It's a useful one. Um, you know, we have found that, um, that MTT U is correlated with, um, security, you know, packages, packages that are better about managing their dependencies, um, you know, have, uh, sort of, uh, on average, less security risk mm-hmm <affirmative>. Um, and, uh, and then it also is correlated with, uh, sort of, uh, time to remediate. You know, if you can, if you can keep, uh, your update time low, then your remediation time will be low for exactly the reason I mentioned before. Right? Mm-hmm <affirmative> that if you're, if you're up to date a lot of times, uh, you're already patched, uh, when a new vulnerability gets disclosed,
Mm-hmm <affirmative> so a great comment from John S synopsis back in my R and D days, 10 years ago, I even made a habit out of picks, picking new XML, parsing libraries for every integration I did to find out which one I found easiest to work with. Did I ever upgrade them? Of course not.
That's a great story.
I wouldn't mind time machine to go and have a stern talk to myself. That's awesome. That is outstanding. Um, so moving forward, you mentioned log for J so managing those large scale security incidents, right. Especially ones, you know, that, that, that make it to the news, and then everyone is scrambling. And then, you know, CEOs who normally might be assuming security now are, you know, pushing things down to chain, you know, give me an update and tell me, you know, what's going on here. Um, what, what was, do you think there are things that certain organizations did really, really well? Were, were there places where you saw some others stumble there?
Yeah, I think, um, you know, that really, uh, emphasized the importance of having this inventory. I mentioned of, of applications, you know, knowing mm-hmm, <affirmative> knowing what you have, um, organizations that could, uh, sort of very quickly run a search, run a query and find out, you know, are we using log for J you know, how pervasively are we using it? Uh, what versions are we using? Um, those organizations had a significant leg up in the remediation, uh, process. Um, you know, if, if, if it takes you several days to even figure out whether you're using it, then that obviously shows you done.
Yeah, yeah. That, that definitely,
And we're not in the security space, obviously cloud BS, but a lot of our customers naturally had that same issue. And I think in our experience, cause we actually had a lot of people asking us, how do you deal with this from a, you know, rolling this out, across all of our applications, like how do we detect this? And really it is a matter of having a good process in place where using the right tools you've got, um, you know, you are looking at that SBO and you can generate these like basically enforcing that whenever you're releasing a new version of your software, you're passing all those gates. You're not, uh, accidentally, you know, skipping when you know that there's some big vulnerability. You're not allowed to actually go past that. And so the companies that already had that in place, uh, we're able to actually get past this and upgrade significantly faster. Cause they did have that bill materials to understand, you know, where they were vulnerable and for the software that they might not have had that build materials for, you know, when the next run and try to release some new software, they can figure out a crap I do have, and this vulnerability mm-hmm, <affirmative>, let's get that resolved. Mm-hmm <affirmative>
And, and I would say, you know, while cloud BS, isn't sort of doing direct security scanning itself, you know, it plays a critical role in, uh, security, you know, because CI the, the CIA and CD process, um, that's a sort of critical point to monitor, um, from a security standpoint, you know? So when you think about, uh, the tooling, that's gonna generate that software of materials for you so that you have that, you know, so there's always up to date, you know, in your inventory, that's probably gonna be run as part of the CI process. Um, you know, policy is also typically enforced as part of the CI or, or maybe the, the CD part of the process, right? Where, um, you then look at that software bill materials and you say, Hey, you know, if any of these things are, have a, you know, medium or above vulnerability, or maybe you said the threshold at high or criti and critical, you know, whatever your threshold is. Um, you know, it's common to have a policy in place to say, you know, we're not gonna allow a release, um, that violates this policy. Um, and that all gets orchestrated as part of CI. And, you know, once it's in CI, then you know, what's happening, <laugh>, you know, it's not something you have to remember to do. It's just part of your process and, and it's automated mm-hmm
<affirmative>. So we have a question from Chris who asks, are there tools that can help you build a software build of materials?
Yeah. So, um, there's, um, you know, so types lifecycle products will, um, will generate us software, build materials. You can export it, you know, in common formats. Um, there's also open source tools, um, out there that will, uh, uh, build, build an S bomb for you. They tend to be, uh, language specific. So, you know, there'll be a tool for Java. There'll be a tool for, you know, for go and so forth. Mm-hmm <affirmative> and so, yeah, the, um, they're out there and then it's just a matter of, uh, working it into your process. Like I said, making it part of CI and making sure that it goes somewhere, right. It doesn't help to generate the <inaudible> and then throw it away. I mean, that's fine. If all you wanna do is enforce some sort of gate, you know, release gate saying, you know, we can't release if we have vulnerabilities. Um, but if you wanna persist that information so that you can quickly respond to mm-hmm <affirmative> zero days, like log for J um, then you need to think about where you're storing it.
Mm-hmm <affirmative> sure. Um, so I know there's a lot of talk about log for J that was about six ish months ago. Doesn't make it less important, but aren't there other, you know, there are other things that have happened since then maybe not getting as much press, you know, maybe I don't know, in stealth mode, uh, or is there, is there anything else recently that's been maybe not as newsworthy, but still pretty significant.
Yeah. I mean, so there was Springs shell, which, um, which did get pressed, not as much as log four J um, mm-hmm <affirmative> maybe cuz it's not quite as pervasive mm-hmm <affirmative> um, although spring is very widely used mm-hmm <affirmative> um, and, uh, you know, what's interesting, there is, uh, we didn't see quite the same sort of rapid adoption and, and sort of as full adoption of, of the patched versions. Um, if you look at mm-hmm <affirmative> we have so, so type has dashboards, uh, live dashboards for log for J and for spring shell. So if you just search for those, you know, log for J Sonotype dashboards, Springs, shell, so type dashboard, you'll find those. And, um, you, it's interesting and fun to compare the two, right? So <laugh>, you'll notice that, um, uh, the adoption curve for, um, the updated version of spring boot, uh, levels off at sort of a, a much lower level than log for J one log for J is now, um, something like 30 to 35% of downloads from Maven central are still vulnerable. Um, oh, which, you know, would be great if that were lower, but, uh, you know, uh, if you look at Springshot at spring boot, um, it's, uh, it's much larger than that. It's well, over 50%, I believe of downloads are still vulnerable. So, um, you know, it, it's interesting and I, you know, uh, it's a great topic of discussion for slack. I'd love to hear people's thoughts on like, why that might be, um, that uptake was so much lower, uh, for spring
Maybe dependency management is not where it should be. <laugh> yeah. I mean, uh, wow. That, that is amazing though. That is amazing. So what, um, that, that is interesting that it is still so high. I, I am actually surprised by that now that you say it, but, but what do you think contributes to that? I mean, we talked about dependency management, but what else, what else do you think may be a contributor?
Yeah. Um, you know, it could be, it could be maybe a perception about the severity of the, you know, vulnerabilities, like the, you know yeah. Um, when people don't update a common sort of rationale is, oh, you know, the way I'm deploying it or the way my application is configured, you know, we're not vulnerable to this mm-hmm <affirmative>, um, and sometimes that's true, you know, like you do have to be careful of spending more time justifying, not upgrading than it would take to just upgrade, you know, mm-hmm, <affirmative> often, um, it is best to just go ahead and upgrade, but, um, yeah. You know, it could be that, um, it could be sort of the profile of the applications, you know, are they, are they sort of in production or are they more internal applications? Um, although, you know, spring is pretty widely used for externally visible applications. So I, I, you know, sort of expect it to be more critical, but, um, mm-hmm, <affirmative> yeah, probably those, they're the two things that come to mind.
Okay. Um, so now we're in, we're in our free for all section <laugh> mm-hmm <affirmative> of the conversation where we can take any questions, any answers, but, um, talking a little bit about, um, progressive delivery again. Um, why do you think, I, I know for some people it is still such a new concept, right? Um, Logan, why do you think there are some organizations that have not yet fully explored it or haven't, haven't implemented progressive delivery?
Yeah, that's a great question. I think it's, it, it feels a bit scary to be able to push and give people the flexibility to push, you know, code into production that mm-hmm <affirmative> is not as heavily vetted. I mean, you would hope that you'd have as many tests and all that in place, but when you're focusing on, you know, increasing that speed of innovation, mm-hmm, <affirmative> realistically, if we're saying, Hey, let's get to production faster, you know, shortcuts might be taken. And so I think inherently, there's a bit of just concern in the back of your head that says, is this gonna be risky? Like, I know I can turn it off, but could there still be problems mm-hmm <affirmative> mm-hmm <affirmative> and I think this <laugh>, isn't helped by the fact that a lot of people probably even cloud B's people, maybe even myself have <laugh> said that, uh, you know, testing and production is like always, always quote, testing and production with future flagging. And it's like, yeah, you want to qualify that by saying, we're testing. Like, is this function good? Like, like, is it converting as well as we want, are people getting value out of it? Right. Not does this thing break or not. <laugh> like, that is not the type of testing in production that you wanna do. And so it's important to make those kind of distinctions. And I think just the fact that it is a fairly new concept, it's something that has just recently built up a lot of adoption. Mm-hmm <affirmative>,
Mm-hmm,
<affirmative> like there's yeah. Some still a lot of misconceptions out there. And at the end of the day, I mean, it truly is same things that people have been doing for a long time. Mm-hmm, you know, before feature flags were, you know, called feature flags, people still did remote config with just those, if statements calling out to database mm-hmm <affirmative>, um, you know, it's, it's been around for a while, but yeah, it's just the fact that this is a cool new thing. I think just still scares people a bit.
I, I think another thing that's important to keep in mind is, um, you know, there's, there's a certain element of like just adhering to this, the level of discipline required to support that process has a positive impact on, on code quality. You know, um, we, we see this for deployment frequency as well. You know, like I, I'm a firm believer that, you know, it's not like, so, okay. The data shows that, uh, PLA when you deploy more frequently, you're generally more secure. You have better business outcomes. There's a whole lot of positives that flow from increased deployment frequency, you know, is it all due to just pushing, you know, more versions of your application out there? No, most of it is probably, you know, the discipline and the processes is required to get to that level of engineering excellence. Right? There's a whole lot that goes into that, that just improves quality across the board. I think the same thing is, uh, probably true for progressive delivery where, you know, if you have to engineer your code so that you can support this feature being on off, you know, there's a certain amount of robustness that you have to build in. Um, and that robustness is I imagine beneficial outside of just that particular feature flag. It probably makes the code better as a whole.
Absolutely. And that reminds me of, of one, uh, point, which is like, you also need to have additional tools in place to support this. Mm-hmm <affirmative> like just being able to toggle it on and off is, is cool. But you know, if you're actually going to be, um, you know, testing things in production in the sense that we wanna see is this new feature successful mm-hmm <affirmative>, then you do need some metrics you need, uh, you need, you know, analytics tools to see are these events happening that we'd expect are happening. Um, you also want to have good server metrics. Mm-hmm <affirmative> are we getting 500, these
Telemetry in there? <laugh>
Exactly. Yeah, exactly. And so I think, as you're saying, Steven, like by adopting this practice, it does mean that you need to have these things in place that you might not have otherwise had. I'm not hoping 20, 22, most people have some form of telemetry and are doing some analytics and all that good stuff, but mm-hmm, <affirmative>, it, it, you know, there's not much value in future flagging if you don't have the right tools in place to actually benefit from it.
Yeah. And I think your point about measuring the right things is important too, right? Like you have telemetry in place, but are you collecting the right telemetry? You know, do you like you, you have a great opportunity to learn a lot from these future flag experiments. Um, but you need to be collecting the data.
We have a question from Eric at what level of depth should dependency be if supply chain in deep, what is the best practice? So, uh, you know, we're, we're, we're doing dependency tracking dependency mappings, but how far down do we go?
Yeah. I mean, you know, <laugh>, in some sense, right? Any, even a very deep vulnerable component, you know, exposes you to risk. Um, it doesn't, you know, it doesn't matter whether you directly call it or you call some library that calls some library that calls, you know, this vulnerable method. Um, and that poses a big challenge for remediation, right? Because, um, uh, you know, you're inheriting that vulnerability, but you're not responsible for the maintenance of the package, that's bringing it in. Um, and so, you know, ultimately, you know, it's up to sort of the community as a whole to make sure that we're all updating quickly so that those transitive vulnerabilities, uh, get patched as quickly as possible. Um, I think there's also, uh, you know, this, this sort of speaks to the importance of being careful about what dependencies you choose, right? So, um, you know, if you pull in a dependency that, uh, is not great about updating their dependencies, then you'll be exposed to that transitive risk for much longer. Um, and so when you're thinking about which XML pricing library to use, you know, one of the things to consider is how on top of, uh, you know, their own dependency maintenance, are they
Mm-hmm <affirmative> <affirmative> for sure. Um, and I, I just think about the nightmare, uh, of, of, of, of working through something like that. If, if maybe you were trying to, uh, manually sort through or write scripts at the last minute, figure out where everything is versus already having those dependencies mapped out, um, that that would definitely be a challenge. Um, what is, um, what are some trends you see, um, uh, around security as far as, um, these aspects of these vulnerabilities? Uh, Steven.
Yeah. Um, I mean, so one trend, uh, that we've seen recently, that's kind of new is, uh, malicious code injections, the idea that, you know, um, it's not all accidental vulnerabilities anymore, you know, there are people working to inject vulnerabilities into popular, open source project.
I've seen that and it is,
Yeah. Yeah. It's scary. Um, yeah. <laugh> and so, uh, they're, you know, that's sort of like, it's, it's a new landscape for the attacker and, and it's sort of, uh, required new responses on, on the defensive side as well. You know, one thing that we've done to help address that is, you know, I mentioned, uh, sort of having a, an artifact proxy, you know, uh, an artifact server, something like nexus or artifacty where you can, um, you know, monitor where you pull in, uh, dependencies, like, um, you know, we have a product called firewall that, that scans the, uh, sort of lets you, lets you know about, uh, dependencies that you're pulling in that seem to have had malicious commits lately. Right. And so you can quarantine mm-hmm <affirmative> you can sort of set a policy that says I'm gonna quarantine this. I'm not gonna allow the use of this new version until the community has sort of vet vetted these, uh, suspicious commits. Right. Mm-hmm <affirmative> um, and so that's one way to approach, uh, that problem and uh, but you know, it's, it's definitely, um, I don't know, it's, it's, it's opened up a new, a new landscape of, of things that we have to worry about from a defensive perspective. So it's important to think about, well, you know, what technology do you have in place? What technology can you put in place to protect mm-hmm <affirmative>
Um, one of the, one of the, I think one of the last questions, uh, that I have, so, so audience get 'em in, if, you know, if you got 'em send them, um, is around the different, let's call it the different types of developers, right? Mm-hmm <affirmative> because you people want to say, you know, developers, the developer, but then, uh, everyone who touches code and everyone who works on a team is not responsible for the same things. I think about how you can have, uh, you know, a platform team that's working on behalf of the whole organization. You have teams that are definitely aligned to certain features and certain, uh, products that they're trying to push out. But one of the groups that I think is coming up more and more often that a few years ago, not as much, um, the SREs, right. So how are SREs impacted with, um, with, uh, the progressive, uh, deliveries there? Is there a benefit for, uh, SREs around progressive delivery?
Yeah, that's, that's an interesting question because you know, I've unfortunately worked with, with companies that have had all sorts of different definitions of SRE mm-hmm <affirmative>, but I'm gonna take it to mean like the actual, like Google SRE, which is like trying to keep up a yes. You know, a massive Webscale application and yeah, I think progressive delivery really can help them from the perspective of we're no longer having to deal with. And I will say most people who have the point, like they're at the point of having SREs mm-hmm <affirmative> they probably do have a lot of good processes in place. Mm-hmm <affirmative>. Um, but you know, anything that helps, uh, help that rollback process when bad things do go into production, mm-hmm <affirmative> because even in the scenario where you have a good Canary deployment, um, where, you know, you're rolling out progressively on the actual server side, not, not the feature side.
Um, mm-hmm <affirmative> you saw the problem that I discussed earlier, where, um, you know, realistically, if you're releasing multiple features at a time, mm-hmm, <affirmative>, you're still ending up in a situation where you're, you know, one, one bad feature is blocking the rest of them functioning mm-hmm <affirmative> you have to roll back the entire thing. So mm-hmm, <affirmative>, I think this really helps everybody involved from the perspective of, you know, developers can get their code out there faster PMs can, um, you know, get information about the new features, the SREs, they don't have to worry as much about, uh, the actual rollback itself. Mm-hmm <affirmative> just instead using the metrics that are, you know, that they're using with their tools. So then determine when it's appropriate to turn those, uh, flags off
Mm-hmm <affirmative> mm-hmm <affirmative> yeah. I mean, future flagging sounds like it could be a dream come true for SREs. If the organization has to processes, align properly. Um, any SRE impacts that you've seen, um, with security, because again, I think most people, when they think of these things, they are thinking just of, you know, the day to day, you know, development teams that are, that are sitting there maybe, uh, cranking out code, but what, what SRE impacts do you see?
Yeah. You know, I think, um, there's, there's sort of this, uh, always difficult question of like what responsibilities live, where, you know, um, yeah. Especially with DevOps and dev sec ops, you know, develop mm-hmm <affirmative> the answer more and more is, oh, developers <laugh> own that one. Um, and you know, even, you know, to the point where like, yeah, it's some amount of SRE falling on the developer's plate, you know, or, um, are there security issues that are best sort of monitored by the SREs like that, that negotiation has to happen? Mm-hmm <affirmative> just to figure out sort of whose responsibility is what, but, you know, there's definitely, like, I think there are advantages to specialization, right? Having people in charge of SRE having, you know, even on the op side experts that, you know, can, can help get things set up, you know, and follow best practices and, and educate new developers. Um, but, uh, there's also, you know, a lot of value in sort of having one person own things end to end. So I, I think the answer for each org is different, but it's a conversation worth having if you haven't had it.
For sure. All right. I think we are pretty much, uh, at time this has been a fantastic, fantastic, uh, conversation. And, um, I, I think there's a lot of value in everything. So we started around progressive delivery, um, worked our way through security, some of those large scale vulnerabilities that everyone has seen, uh, really doubled down on dependency management, dependency management, dependency management, and just do it, you know, there, there are four <laugh> four letters there we're only gonna use three. Um, and, uh, and just how, you know, uh, comprehensive this is, especially if you have a very large enterprise, um, any parting thoughts, um, in, in the last, uh, moment, uh, that we have here.
Yeah. I think just in general, um, there's always a lot of cool new processes and, you know, techniques out there mm-hmm <affirmative>, and it's always just important to consider, you know, is this appropriate for our use case? I think progressive delivery is an amazing new way of delivering software, if all the kind of key things align like I talked about earlier, and I think it's just worth exploring those as they come out. But, um, you know, you don't have to go all in. You don't need to make a, a big decision of like, this is how we're doing everything from now on, you know, just experiment and, uh, see what works
Mm-hmm <affirmative>
And, you know, happy to chat with anybody after this, if you wanna talk more about progressive delivery
Mm-hmm <affirmative>
Yeah. I, I, I would say also, you know, E echo that, right. You have to do it, sorry for your organization. You also have to consider, you know, the people in processes as much as the tooling. Right. Um, it really starts with getting the right culture in place and the right practices. Mm-hmm
<affirmative> excellent. Thank you again. Thank you audience for being so engaged, great comments, uh, and great questions and, uh, wonderful vendor dome. Thanks again.
Thank you. Mm-hmm
<affirmative>.