Saving the Software Supply Chain: Critical Trends and 5 Must-Dos for DevOps (US 2021)

DevOps toolchains and processes have become a preferred entry point for bad actors to infiltrate the software supply chain. As a result, DevOps professionals are becoming lead actors in protecting against software supply chain attacks. In this session we’ll share the latest trends in software supply chain security and share five actionable must-dos for DevOps teams. Attendees will come away with a solid understanding of the current and emerging threats facing software supply chain security and tips to help any organization secure their development processes and increase the security of their DevOps practices. This session is presented by Anchore.

usbreakoutlas vegasvegas2021

Daniel Nurmi

CTO and Co-founder, Anchore


Kim Weins

SVP Marketing, Anchore



We're excited to be here today to talk to you about saving the software supply chain, critical trends, and five must use for dev ops. I'm excited to have here with me today, Daniel Nurmi who's our CTO and co-founder at anchor and I'm Kim wines, the SVP of marketing at anchor. All right. So we'll, that'll be the gap. Let me go ahead and present. And then I'm going to go over here, share the screen, put this down and maybe the lower corner. So it won't be obvious. All right, so we'll go ahead and continue all intra. Well, do you want to just pick it off? I don't need to intro you. Why don't you, why don't I just go to the next slide and then you just start right here.


Okay. That sounds good. I'll start, uh, right now. All right. Thank you very much, everyone for being here. We're going to start this presentation by talking about the question, why focus on software supply chain, right. And it's a little bit about a definition of, of, uh, of what we're going to be presenting here. Um, what we're looking at here, it really is a depiction of a common, you know, uh, CIS CD or development, uh, infrastructure that you might have in place. It's might look familiar where on one side of this picture, we start with our application source care. And then in the middle, we're going through a number of stages of taking that source character to get into executable walls, doing build pipelines and things like that. We're at the very end, you know, the output of this kind of machine is we end up with something that we're, we're publishing software, we're publishing and offering out to our consumers and customers, or we're taking those applications and executing them in production.


Now, underneath that, we've also got our platform. This is all of the infrastructure that facilitates this whole process, our source code management systems, or CICB systems, automated QA and testing, and finally things like our publication systems or our, our orchestration and execution environments. Now, when, when we look at it, usually as software producers, we just look at, you know, this is a machine it's functional. It's doing, um, uh, a lot of work for us. When a malicious actor looks at this, they see a lot of interesting areas where they might be able to come in and infiltrate this environment in order to influence your software injecting malicious code, or putting in a bad password or, or escalating privileges, that kind of stuff. And so when, when we look at this picture from, from a security perspective, we also have to think about work for all of these stages.


These are the kinds of areas that are potentially weaknesses that could expose us to a security attack. And generally speaking, when you're talking about supply chain security attacks, these are the kind of areas where hackers will come in and make some, um, you know, some, some attack, um, uh, happen in order to get their code introduced into your code or your infrastructure. And then your software, uh, is compromised so that people down downstream consuming your software or software is executing, has that, that, uh, malicious actors software or, so this is kind of the general construct that we're going to be talking about today. And next, I'm going to pass it back over to Kim. Who's going to talk about a survey that we performed, um, asking folks, uh, various questions about the topic.


Thanks, Dan. So one of the things we wanted to do today is really bring some data to bear, to look at what are organizations doing today and how are they concerned about the software supply chain and how are different practices in their organization, um, either helping or hindering in cyber supply chain security. So the survey we're going to be referencing today was a 400 and twenty-five technology leaders at large organizations. So that means organizations with at least a thousand employees. Many of them were at much larger, larger organizations over 5,000. And this was CIO, CTOs, CSOs, VPs, directors, managers, and architects that spanned across dev ops development. It security and cloud across a variety of industries. And we're focused in two regions about three quarters in north America, and then about a quarter within the European regions. So this is really going to give you a picture of what's happening in those larger organizations in terms of software supply chain.


So one of the interesting trends that we saw is not surprisingly growing container adoption, but also that grind container adoption is creating new security challenges. So this data shows us that a quarter of the users that we surveyed fell into a category of advanced container adoption, meaning that most of their apps were in containers. And then another 40% fell into the intermediate category where a significant number of their apps are in containers. So together that really represents about two thirds of the respondents were really in those, you know, middle to later stages of container adoption. Now, as we ask those different users at different levels of maturity, how they saw a supply chain risk. And the interesting thing here is that the more advanced they were in container adoption, the higher their perception of supply chain risk. That's really interesting because normally you might think this would be in reverse, you know, in the space. For example, we saw that as users use cloud more, and they got more comfortable with the tools and the technologies, what the cloud providers were doing, they started to worry a little bit less about security. They knew that they could solve and address the security issues here. We're really seeing the inverse that as people start to use containers more, they start to recognize that there are potentially opportunities for threats to come into their supply chain, into their software spec factory, so to speak into their dev ops process.


So Dan, maybe talk a little bit about what you think that means for dev ops teams.


Yeah, absolutely. I mean, these are very interesting responses to Kevin's point and you know, this kind of leads us to, you know, what we consider a must do, uh, which is really around the idea of we as practitioners, as software providers, we really need to create specialized processes that actually understand containers. And the reason for that is that, well, first containers, uh, represent kind of a new type of software bundle that some of the more traditional security tooling, uh, isn't aware of containers can have yes, the application that you're building and the container kind of usually has a name. It is an application or something like this, but it also can typically have almost an entire operating system environment. That's additional software, additional configuration, um, other stuff that comes in, in support of that executable application containers can also have data artifacts that can have, um, you know, secrets and keys, anything that's required to support the execution of that application in an encapsulated container like way are also, uh, elements that are part of that executable.


So development processes built around containers in addition are typically built with the idea of a high degree of automation, uh, in mind. Uh, and this leads to in your process, oftentimes a lot of containers being generated, whereas previously you might've had far fewer executable artifacts. Now with containers, we might see a lot of containers, a lot of things that can actually execute or be distributed to your clients and customers. That's another interesting new thing about containers and all of this really leads to the fact that when choosing tools are looking at, you know, how to change your processes, to deal with the fact that you've got containers with these new characteristics, you really need to make sure that our, both our processes and our tooling understand what a container is, what are the security implications of having them and how to, how to get that information and then secure them properly.


All right, great. Let's go on to some additional trends. Um, so the other thing we want to highlight is that cloud native applications often built with containers, but, um, use can code from multiple sources and this shouldn't be a surprise to anybody in the dev ops world. Um, we asked people what they estimated to be the, the source of software in their containers, and they estimated that about a quarter of that code was open source. Now, in fact, we've also seen data, um, from scans of software applications that this made in fact be much higher than in many industries. It can be from 60 to 80% of source code is represented by open source, but that's something that organizations are really learning how to grapple with how to manage and how to secure. Now, given that open-source is such a big part of source code in modern cloud native applications, uh, we ask people about their top container security challenges, and you can see here at the top of the list is the security of open-source containers. So 23% of the respondents ranked that as their number one security challenge, um, followed by security of the code that we write. So looking at the fact that open source is representing more and more of your code and that people are challenged by open source security. That really leads us to our next step ups to do.


Yeah, this was another interesting finding, um, not necessarily as closely related to containers, but definitely related to the, the, the, um, the, the concepts of supply chain security. And, you know, we, we kind of think of this as, uh, boiling down to the statement. You really need to pay very close attention to software that is not developed in house, right? So the, the, the critical, um, I think, um, you know, concept understand is that while we, as a software development organization or any organization, that's got some software development in it, we understand our own applications very well. And we think of what we're producing is our application, but in reality, most applications today really do depend on, uh, quite large and rapidly updating collection of external software coming in from a variety of external sources, all of which have their own security levels and processes involved. And so the real takeaway here is to, um, from our perspective, it's really important to differentiate between the kind of security controls you're applying to your own application source code and processes, um, from the security implications to the wing reports, policies that you're generating around all of the external software that you're bringing into your environment in support of your application.


All right, let's move on to some further trend data. And this was really finding that diverse dev ops tool chains are definitely going to require security attention. So we asked people about the CIA and CD tools they were using in their organization. We wanted to understand whether there was diversity of tools, and we found that there absolutely is only in 20% of organizations. Was there really a single CICB tool that had been selected and was being used by all teams in 38%, there might be a preferred CICB tool with exceptions. And then in the remainder of the organizations, there was either multiple tools or people were just selecting their own tools, meaning that it was a free for all. And I think that's because, uh, development organizations want to be as efficient as possible. And so they want to pick the tools that they're comfortable with. Also, as organizations grow and acquire, often neutrals are brought in to add to existing tools. Now it wasn't really just about the CIC D tools, but beyond that, when we started to ask about, uh, things like registries and other repositories, as well as CICB tools, et cetera, we found that on average organizations were using six different medium, a median of six different dev ops tools. So that means that there's a lot of different tools that exist in the organization that all need to be secured as part of your software supply chain security initiative.


Yeah. And this was another interesting one because it definitely reflects, you know, what, what a lot of organizations already know, which is that there's a lot of variety in your, your choice of infrastructure tools to facilitate that, that SPLC or that development life cycle. Now from a security perspective, you know, something interesting has really come out of here. Um, uh, this observation at this moment in time, which is that we're seeing more and more of these types of infrastructure platforms are building in more security controls into their own platforms, which is definitely a good thing. However, um, oftentimes those security capabilities are really matched to the platform itself. And so it's good that there's more security controls built into a lot of these platforms, but it's also complex and create some problems operationally, and that if you have five or six different CICB systems, each with their own security tooling, it's somehow sometimes hard to kind of gather all that back and get a consistent policy across all of those tools that are providing slightly different reports and capabilities and functionality.


So from our perspective, it's okay to have multiple tools, um, uh, all providing security scanning, but it's really important to have at least one type of tooling that can sit on top of your entire environment and be able to aggregate all that information, do its own scans, even if they're overlapping, because, uh, there's certainly nothing wrong with having multiple security tools, uh, checking for the same thing. Um, and in addition, that platform agnostic approach can actually get you a benefit that sometimes, um, uh, we don't think about it as much, which is that it provides you a little bit of, uh, insurance against your platform being in foster infiltrated that can also lead to your security tooling, being infiltrated if you're using the same, uh, uh, security, uh, capabilities of the platform itself. So by externalizing that you give yourself a little bit of, um, uh, risk reduction, uh, for getting your security, um, uh, uh, uh, value your performed over for all of all of those environments.


All right, great. Let's move on to the fourth trend that there is room for improvement and securing the software supply chain. So we asked organizations whether they have been affected by a software supply chain attack in the last six months. And you can see that about 60%, um, actually 64% were impacted in some cases that was a very significant impact. Some cases moderate in some cases, a minor impact, but really this is starting to, uh, hit most organizations out there today. And we expect that to include an increase or continue as time goes forward. Um, so we also then wanted to understand, okay, if you're being impacted by attacks, what type of supply chain practices have you implemented already? And specifically, we were asking for containers and you can see just about half are scanning third-party containers and open source containers. And then from there, some of the practices really go down where very few people are a very small minority are implementing them. And I want to especially identify the things that are at the bottom of the list, which are about S bombs. So as farm stands for software bill of materials, this is starting to become a way that people are recognizing as an important aspect of being able to, um, help to secure the software supply chain. And very few organizations are either requiring an S bomb from their software suppliers who are sending them software. And also very few are producing S bombs for any software that they are creating. So that leaves an opportunity for some improvement.


Yeah. And, uh, th this, this is, uh, know it leads us to our next, uh, must do, which is, you know, uh, from what we're seeing, um, in, in the security space, when it comes to the conversation around supply chain security and a number of other things that I'll discuss in a minute, um, we've, we feel that it's time now to really get on a path, to start producing and asking for software bill of materials from your software providers and as a software provider to provide that same kind of, uh, uh, metadata. And we're seeing this more and more organizations are starting now, as we saw in the, in the response, it's a, it's a small percentage relative to some of the other responses, but it's not zero, um, uh, organizations that are actually asking and requiring that their incoming software include alongside the software itself.


This rich set of information about all of the other software, including open source dependencies and vendor dependencies and configurations and proof of, um, you know, security practices, licenses, rich metadata that comes along with a downloadable, uh, essentially, um, uh, that, that practices is only going to continue to increase. And this is, this is something that's been manifested pretty pretty, uh, concretely in the U S executive order that came out, uh, earlier this year, uh, which is really moving forward at a, at a fast pace. And we look at that as a very good example of, um, this practice starting to become a reality where that order actually stipulates in order to provide some software to, uh, certain, certain government agencies. It must come with an S bomb and a number of other materials alongside the software. So we're gonna, we predict we're going to start seeing that happening, uh, more and more even between enterprise organizations and other organizations and projects between open source projects and organizations. And so, um, we think that now, uh, it's, it's a really good time, and it's actually pretty straightforward to start, including in your software development practices, because the generation of these, these, uh, software bill of materials documents, anytime you're building a new new application software.


All right, great. A final trend. Uh, what challenges are organizations facing with container scanning? So we asked respondents, what were the top container scanning challenges? And you can see here we've differentiated between a significant challenge versus somewhat of a challenge. You know, the number one is probably not surprisingly identifying vulnerabilities in containers. Uh, but then it was interesting what came up as number two, three, and four, which was about getting developers to remediate wasting developer time on false positives, or it taking too long to remediate. So these were really about the issues that developers are facing, or organizations are facing as they try to not just identify the vulnerabilities, but fix them. We also asked about false positives specifically, because this is something that we've heard about being a significant problem. I think we're all aware of that, that have been working in, in securing software. Um, but we wanted our respondents to estimate what percentage of vulnerabilities that are found, do they feel are false positives and really almost half 44% they feel are false positives. And so that really, uh, raises some questions about really, how do you manage and deal with that when we know that in the, in the CVE data and also in the nature of the scanning, that there can be a lot of false positives.


Yep. And this, this one kind of leads to our final, uh, most dear recommendation, which is actually oriented a little bit around the notion of left. And so before I talk about that, just briefly about the topic of false positives, you know, going all the way back to some of that information, as I'm going over earlier about containers and how they're composed, there's a lot of software inside of a container. Um, and, and, you know, as we know, there's lots of open source dependencies, the vulnerability data, um, being published out there is, uh, there's a lot of it. It's not all on the same form. And so it's, it's sort of a reality that with a scanning tool, that's looking specifically for software vulnerabilities, we need to live with the notion that there will be false positives, um, or, uh, almost equivalent to an actual false positive is a vulnerability that exists and is accurate.


But when our security team does an assessment determines that it, it's not gonna impact the application, right. Which is just as expensive as evaluating false positive, maybe even a little bit more, but we're going to need to, to sort of live with that reality, that there will be a constant flow of, you know, software and vulnerabilities reported against that software. And so the must do here really is about the process for dealing with that. And in our opinion, shift of security approaches are really unimportant way to try to address and handle this reality a bit more efficiently. The reason for that as well, first of all, shift left is the notion of making sure that through all of the stages of your software delivery life cycle, you've got some security scanning. You can look for as many things as you can at every stage, as new softwares can introduced towards that stuff, being executable, um, do that early and often throughout the entire software delivery life cycle, uh, we kind of consider that a shift-left security approach.


And by doing that, there's some real benefits. You'd get the obvious one in a sense is that the earlier we're able to find issues, um, the more value valuable it is to be able to detect that because the later in you go in that stage, the closer to exposure you're getting. So at the extreme, we've got an application it's actually running in production, finding a security problem at that side is far more expensive than if you're able to find it just prior to execution, for example, but the other value that may be a little bit more subtle and really maps more to this idea of making the process of dealing with false positives, more efficient is that the sooner you can find a security violation and we're finding, uh, the closer you are typically to the team or the person or the process that has the ability to remediate it, no, rather than looking at a running system and having to go all the way back to a developer, if you're able to find the security violation right after the code has been committed, you're very close to where that developer, there is a person who can fix the problem, um, uh, lifts in terms of information flow and ability to get that done quickly, far more efficient, to be able to find these things as soon as possible.


And so that, that really leads us to that that must do of implementing shift-left processes. It's far more efficient and valuable when it comes to security or your chain.


All right. Thanks, Dan. And thanks everybody for joining us for the session today will be available on the slack channel for Q and a, and we shared some results from a survey. If you want to get the full results of that survey, feel free to go to, and you'll be able to get all the trend data, uh, to help with your dev ops process and to help, um, make that more secure to protect the software supply chain. Thanks everybody


Very much.