VendorDome: Software Supply Chain Attacks: Why They Matter and What DevOps Needs to Do (US 2021)
With both cybercriminals and state actors focused on infiltrating businesses and government agencies, software supply chain attacks have become increasingly common. As organizations respond to threats and work to protect the software they consume and develop, it can be difficult to know where to start. In this session, experts from Anchore and Aqua Security discuss the actions that can be taken to combat these threats—and what DevOps professionals in particular should be doing to improve software supply chain security. Attendees will gain a seasoned perspective and insights to the latest industry trends and technology. This session is presented by Anchore and Aqua Security.
Daniel Nurmi
CTO and Co-founder, Anchore
Rory McCune
Cloud Native Security Advocate, Aqua Security
Rani Osnat
VP of Strategy and Product Marketing, Aqua Security
TRANSCRIPT
Hello? Hi everyone. Uh, this is Ronnie Osnat VP of strategy and product marketing at Aqua security. And I'll be the moderator today for our session. I'm very pleased to have with us, uh, an army, uh, CTO at Accor and or a cloud native security advocate at Aqua security. Hi guys, why don't you go ahead. Introduce yourself, Daniel. You go first.
Hi. Uh, thank you so much for having us to talk about this important topic today. My name is Daniel Nurmi, I'm the CTO and co-founder of incor and we're a company that was started, um, around the time when containers were really starting to pick up steam. And what we do here is produced technology open source and products that really aims to help organizations who are adopting modern development techniques, um, to, to introduce security compliance, best practices directly into their, their dev ops tool chains. So very pleased to be here. And I'll be talking to you today.
Thanks for joining us and, uh, Rory.
Oh yeah. Uh, my name is Rory McCune. Uh, I'm a cloud native security advocate at Aqua, and we are essentially focused heavily on the cloud native and container ecosystems and really helping customers as they go through their life cycle, moving more into cloud native and into containerization and secure that entire process. And I'm really glad to be here to talk about this as it's an issue. That's, I think attracting a lot of attention to the woman,
Right? And today our topic of discussion is software supply chain attacks and, uh, why they matter and what dev ops need to do. Um, and you know, this is, uh, something that is obviously taken a lot of, uh, uh, attention received a lot of attention recently. Um, why, why is there a focus on the supply chain right now? Um, I guess Daniel, you start.
Sure. Uh, you know, th this, this is a, as Roger is mentioning, you know, really, uh, topical, uh, you know, uh, discussion that's been going on in a lot more focus, especially over the last six months to a year. And I think part of the reason is that, well, there's been, you know, simply some high profile security events that have happened, right. Um, you know, things around, uh, you know, uh, code Cub and solar winds and, and other types of very high profile, large impact, large blast radius, security events. And, and a lot of that, you know, the discussions after the events started to ask the pertinent questions, you know, what happened here? What what's happening? Why is this something, you know, uh, that that's, that's starting to affect so many organizations when the actual impact or the actual event happened in a very focused location.
And some of that discussion is really started to formulate around this topic of supply chain security. And so, first of all, I think these hot high profile events really start to focus the discussion whenever they happen. Uh, but in addition, I think a lot of this has to do actually with the adoption of modern development infrastructure techniques, where there's a lot more automation, there's a lot more software that's actually being combined to produce applications that ultimately run in production. And the industry started to really focus on all of the processes that are used for automating a lot of the development and delivery of applications. And trying to figure out with that automation, there's been sort of an increase of the amount of software, the dynamism of these types of systems and how it's all coming together. And we really need to focus on that process of building these applications, because that's part of a more general global supply chain, and that's where these events have really been focused on. So I think that's why this is a topic of discussion right now,
Right? Anything to add to this?
Yeah, I think there is, I think it's a good point that we are seeing a lot of this. No, but this has been building for awhile. You know, this is this move. This change has been, I actually did a talk up on this kind of topic six years ago, uh, north, perhaps at conference looking at, um, things like NPM and Ruby gems and how those package supply chains were secured. And so this to me has been something that whilst it has hit the prevalence now, like a lot of security incidents, I think a lot of security people probably were thinking this is going to happen sooner or later. I don't know, kind of maybe warning about it. I'm talking about some of these topics. I mean, actually I actually love reading, prepping for this. I went back to a quote and the very first time, I think it was mentioned 1984, Ken Thompson said you can't trust code. You did not totally create yourself in reflections on trusting trust. And so this is like a law school. This has been going a long time, but what we're seeing now is, as Daniel was saying, just the huge adoption has led to inevitably attackers follow adoption, right. That attackers will follow the process of this is where people are. This is therefore the attacker scope. This is where I want to be, and then we get incidents. So yeah, I think that's where I see it coming from.
Right. And, you know, we've seen, uh, very recently there was an executive order, uh, from the, you know, in the United States. Um, and, you know, given this, you know, what did we see? How do we see the open-source community to responding? Um, what's coming down the line and how will this impact, uh, DevOps will be as we stand today.
Yeah. I think, I think this is an interesting one, right? So, uh, sort of building on what, what Gloria, was this talking about, right. This not being necessarily like, you know, a new, um, uh, topic, but there's, there's been so much more software that, you know, the, one of, one of the events that came out of a lot of the security incidents, you know, over the last year, year and a half or so, was this us, uh, executive order where it's essentially a, a laundry list of the same type of security and compliance, um, mandates that the security industry has been talking about for, for a long time, but it started to kind of focus it in and in codified and actually, you know, recommending some, some actual requirements, uh, for us federal projects to be able to, um, or to actually be required to ask for information and proof of security process from any software projects coming into to their projects.
And, and, and so it's almost defining a little bit more of a formal way to describe, you know, as there's a whole community of software producers as a software consumer, these are the kinds of things we're going to start requiring from the software producers in order to add to the assurances that can be used to then assert some level of trust. Right. So I think it's kind of interesting that that's, you know, that that's almost like a formalization of a lot of topics that have happened and it's, uh, it's, it's a good kind of flag in the ground for, for a supply chain security discussion, because it starts to sort of formalize those requirements, you know? And I think,
Yeah, sorry, I thought you were done, I guess, you know, I've, my perception has been that they're, it's kind of, they're catching up to some of the things that have been happening rather than be rather than imposing something that's entirely new. They're taking a lot of the best practices that are out there and kind of implementing them as a, as a best practice, uh, at the, at the federal level, I guess. Um, and, and, and Roy, what do you think, how will this impact several, there are several initiatives happening in the open source world. This will impact them.
It's, it's interesting because it has been driving, I think, conversations that were already happening. So things like the CNCF cloud native computing foundation had a white paper already in draft or in supply chain security before the EO came out. But obviously this ties that in and it gives greater impetus. Another thing we've seen off the back of the executive order is big tech companies like Microsoft and Google stepping up with millions of dollars and saying, we're going to start putting this money directly into, um, initiatives around this problem. Would they have done that without the executive order or possibly, but I think it's fair to say they did this after meetings with the U S government. So, you know, this is essentially saying, well, you know, you companies are making a lot of money out of this. You know, you need to also give back to the security.
So I think that's good. The only place I think, where we see or what I see in my head, a bit of risk with a regulator stepping into this world is it's quite hard to marry regulation, which moves quite slowly with the very fast paced cloud-native landscape. So there's always a slight concern that they'll end up coming up with things that are quite complicated and quite hard to implement. But hopefully if we're taking advice from people at the open SSF and the CNCF that will happen, right, we will see people who've got more experience on the ground, driving the detail and letting the regulators handle the overall overarching piece. So I'm hope I'm hopeful. This will help.
Yeah, I think by the way, this is a, just an administrative note is a good time to tell people that if they have any questions during the session, you know, we're trying to make this interactive. Um, you could put this in the slack channel, ask the speaker track for, um, and we're monitoring this and we'll engage with these questions, um, as, as we go along. Um, yeah. Uh, I mean, do either of you think that there is a risk of, you know, sometimes when the, especially when the federal government gets involved, things can get a little crazy and not necessarily the right direction. Do you see that as, you know what I mean? Do you see that as a risk of, you know, people starting to throw money at things that are not really effective and just, um, uh, trying to do things to satisfy some check boxes instead of addressing the issues in a deeper way?
Yeah. I think that's a fair sort of general, um, uh, comment. Um, however, in, in this case, you know, having looked at that executive order kind of read it through it and try to understand, you know, what the, what the implications are going to be. I mean, you know, it's a pretty dense document and there's a lot of verbiage in there, but I think what it really boils down to is, you know, and, and again, it's a little bit of a formalization around look, you know, as a software consumer, we're going to be asking for software bill of materials, we're going to be asking for some kind of proof of that, that does the producer of software is doing some security practices. They don't go so far as to like extremely at a fine level dictate exactly what those things look like. And in a sense, you know, as anybody who has been a software producer, even selling software through a large organization, you know, oftentimes as part of that, uh, that negotiation or that interchange there's requests from a large organization for exactly the same kind of information, sometimes a big enterprise will ask for, we need to see a spreadsheet with all of your open source dependencies.
We need to see, um, you know, uh, a report of all the security practices that your development team, uh, follows and what compliance standards does your company conform with that kind of stuff. And so it's, it's very similar in my mind. And so I don't see there being a huge risk in this case. Yes.
Yeah. I think I would, I would agree with that. The only concern I have is, you know, there tends to be a, uh, when high level standards come in, there tends to be a process that then that goes down to someone says, I need an audit guide. You know, I need something I can send out with a group of people to your company to assess. And it's at that level, I, there's sometimes a risk that when people translate that from the high level down to the detail that they start putting in these quite bureaucratic sounding, you know, there's a temptation to try and make security perfect, which it never is and never will be. And if someone tries to make a standard that says, do these, do this, and you have perfect security, it will end up like 1500 pages long and almost impossible to comply with. So I'm hoping they will resist that temptation, but at the moment, you're right. I don't see anything yet, but history tells me that it's possible. And let's hope that doesn't happen this time.
Th there's a question here from, uh, uh, the audience about why code cup didn't, um, get more kind of large, larger media coverage, that incident,
You know, that, that for me is, it's an interesting one. Um, security incidents come faster and faster these days. And, um, you see the new cycle churn so quickly that, you know, it's literally a week and then the next thing takes people's attention. Cold cough was, yeah. I mean, it was really serious and I got that spark of attention, but yeah, unfortunately, probably the next week, something else came along and moves out. And, and it's something that I remember that when breaches first became a big thing, people said, oh, when we see these breaches, we'll see companies take things more seriously, but now breaches are every week. And they're just part of doing business. And, and let's be honest, no one pays attention unless it's truly vast. They're just part of the cycle. So I'm hoping supply chain won't go that way. Um, but when you see things like code club, I think that's a good example. I would have thought it would have the conversation with lasted longer, but it did drop out quite quickly.
Yeah, I think so. Do I think there's probably some fatigue after a while there's only so much, oh, another supply chain effect. I mean, you know, there's so much people can ingest and in a short time, a period of time, um, speaking of that, I mean, what do you guys think of, you know, kind of educating people about the risk. I mean, we, before we get into the best practices and what can be done, what are we really protecting against? What what's the, um, you know, what's, what's the worst case scenario, what's the likely scenario, um, in terms of supply chain that would reach us.
Yep. I can do so for me, that's a real interesting one because it comes down to threat model and saying, you know, what is the threat model of your organization? Who does you think you'll be targeted by? Um, because that's going to drive the kind of supply chain attack. You'll see, everyone will see the kind of the automated, everyone will see the kind of, you know, we're spraying every system on the internet trying to compromise things, but there's the piece of, you'll see things that are targeted to you, but then the really tricky one for a lot of modern software is who are your customers? So if you have one high-profile customer, the attackers might go for you because they're, you know, that they know that you're in the supply chain of someone who's high profile. And we've seen that we've seen hosting companies compromised just because one of their customers was a target. And so that's where I think that, that, that it can be a bit tricky to say which ones we need to worry about because going to be for me is a threat more than conversation, but it's not just, what's my threat model. It's my ecosystem's threat model. My, my customer's threat model.
Yep. I absolutely agree. And I, and I think it's an important, uh, element when, when we're talking about supply chain and like, what, what we can do about it is to understand whether or not we're kind of playing the role of a software producer, which to Lori's point, you know, that can can mean you, you might not consider yourself necessarily a target, but if your customers are, you are absolutely a target. So we have to figure out where in the global supply chain, your software element might live versus as a consumer, right. Trying to figure out well, I'm sort of at the end of a supply chain, and I need to figure out how to, you know, what to ask for and how to understand all of the software coming into my environment. And so some of these concepts are a little bit recursive, but, you know, I think it's important to, to try to position ourselves in kind of one, one area or another.
Yeah, I think that's a very good, that's a very good insight. I mean, ultimately, you know, I, I, once I worked at a company that sold software to the automotive industry and then the automotive industry, they have real supply chains, not software supply chains. Um, and you have, you have the automotive manufacturers that have tier one suppliers and the tier one suppliers have tier two suppliers and the theater to suppliers that tier three suppliers, and they have tier four sub buyers basically, you know, create the screws and the bolts, um, and it all rolls up. But the big difference is that every link in that supply chain is responsible and takes ownership and their, their brand equity is hinged on quality, um, that they delivered to the next one. Whereas in software, everything just flows through, uh, right. I mean, if I'm, if I build commercial software and I'm using, um, an open-source package, uh, or an opensource project embedded into my software and that open source project uses some sort of, you know, Java library and that library has a severe vulnerability.
That's basically that then, you know, infects multiple supply chains. Nope. There is nobody stopping this early, earlier on, right. Nobody's a quality controlling the screw, so to speak. And I think that's, I think that's one of the areas of risk that, um, where, um, people don't understand that they're not just responsible for what they're using, but for whatever component they took, what, but that component is using and what that component is using and so on and so forth, it could go back quite a few steps, uh, which is I think the major challenge, especially with open source, um, which brings me, um, well maybe I've, we've gotten ahead of ourselves. Why is open source, uh, so important, uh, in this discussion what's unique about it except what I just said.
Yeah. I mean, so it's interesting. I think you kind of touched on some important ideas that, you know, first of all, you know, if we, if we put ourselves in that sort of, uh, mode of thinking where let's say we're a software producer and we have expertise over our own application source code, we have a development team and, you know, we're producing some, some software. Um, it's safe to say, I think across the board, uh, it's, it's extraordinarily rare that all of the source code, uh, involved in your application is, is source code that your development teams produce, right? Pretty much every application thing is running in production today is going to be having a fair number of not a very large number of open source dependencies that are being brought in, in support of the application. Right? And this is, this is a challenge in a sense, because there's a lot of different software elements, but it's also, there's an opportunity, right?
Open source by nature is highly transparent. And so there are opportunities for us to actually understand not just what open source projects we're bringing in, in support of our applications, but to actually inspect that source, like look at it, try to make some assurances around its provenance and, and understand its security processes. It's all very transparent. And so I think it's important to sort of look at open source from that perspective of not just a problem, but you know, something of an opportunity, because if we imagine the case where we're building the application and there's only closed source dependencies, it's actually a lot harder to understand a lot about those, those, um, those sort of closed source dependencies. So I do think it's important. We have to understand that total, some set of software that's being used to build our applications so that we can pass that along to our consumers. Uh, but at the same time, it is possible to do, uh, because, because of the nature of opensource.
Yeah. So, yeah, I always, when I talk about open source, I always talk about open source as being potentially more secure. So the potential is there for it to be more secure than closed doors, just as Daniel said, because you can look at it. The challenge is of course getting people to do the looking. Um, I, I'm not as aware. I think that there is a, a big challenge to be, um, to be taken on just because of the sheer weight of dependencies. I mean, you can take a non-security issue like the infamous left pad, where one tiny JavaScript library got removed from the supply chain and a whole load of applications in the node JS world had serious problems. And that application just left padded text, but, but it turns out it was a critical part of a supply chain. And so that's, I think the, for me, the element of open source where it's just so large, and this is another part, but accepting security, won't be perfect.
No one's going to code review. All of the dependencies would feel fairly safe saying no one's ever going to do that. Um, if you look at one project, a good example, Kubernetes, if you list its dependencies, including all the three of them, there's about two and a half thousand. So no one's going through those two and a half thousand software projects and checking them all. Um, and I think managing that complexity and coming to a point where you say, how do we as ecosystem trying to identify the key points and make sure those are getting the review we want is, is a challenge, but one that you're starting to see people address. And again, open SSF, the money that Google has just announced around, you know, using that for our security activities in open-source projects. So I think this is good because we're starting to see motion. And that's one of those times where you say, um, both breaches are bad or they're always bad. This ones, these ones seem to at least be spawning some good activity off the back of them because all the sources are at inevitably large target and needs to be addressed.
Yep. Um, I think in, in cloud native, we've seen some surveys that talk about 70, 80% of the code base being open source and only 20% being custom working code. And so it's, and now it's not only a percentage thing, but also open source is open. So there are more known vulnerabilities that are discovered in that then custom code, which could be very idiosyncratic. So I think that's another, another reason to focus on that. Um, so how do we, you know, where do we go from here? How do we solve this, um, dependency, trustworthiness, um, you know, what, what can we do to, uh, what, what kind of controls can we have in place to, to do that?
Well, yeah, so th there's a couple of areas I think that we can definitely do. And there's, there's areas where, um, you know, there's going to be high end stuff. That's great, but, but, but there's an element of what's practical. Like what can people get started with? And there are good projects now for doing things like checking dependencies and checking versions. Um, there's a good project from the honest SF called scorecard where it, it will assess an open source repository and get hub against things like, you know, is this an active project? Is it getting patches? Does it have a security policy? These are fairly basic things, but they're a good starting point. And if we can get the industry by using these tools, unlike, you know, running them, if we can get people to start saying, right, well, every project will have a security policy. Every project will check its vulnerabilities to make sure it's maintaining its dependencies. That kind of is a good place to start. Uh, and, and those are the things we have to do as well, but just starting there, you know, rather than getting put off by the massiveness of this problem, just starting with quite simple activities, I think is a good place to go.
Yep. I agree with that. And, uh, there's, there's also, you know, off the back of some of these, uh, these, these new investments, buy, buy, buy big, big companies and big open source organizations. Um, you know, there, there's some really interesting, uh, you know, more, more sort of, uh, architectures for considering what supply chain security looks like and what kind of practices generally need to happen. Like the salsa project, the alpha omega project, the Linux foundation, and open SSF. I've been doing some of this as well. Microsoft just announced, you know, their, uh, uh, another supply chain, you know, infrastructure sort of, um, you know, uh, design, uh, which is, which is pretty high quality, right? So a lot of these things are just kind of going over, what are the practices that everybody should be considering? And if we all do it, then the global supply chain is going to be elevated, right?
Fewer incidents, you know, um, a lot more transparency, uh, that kind of stuff. And I think no one of the, the sort of key things that that's at the foundation of almost all of these projects is, is really around inspection or insight to the transparency of what is actually there in, in, you know, in, from, from our perspective, working with organizations who are trying to secure their development processes, a lot of the security teams that we work directly with within big tech organizations, at least, um, they require information, right? It's not necessarily first about the security tooling itself. It's purely starting with what software is there. So, you know, being able to figure out just fundamentally of all of the software that's coming into the organization, stuff happens in the middle and as we're building stuff, and then we deploy software, what are all of the software components that make up our entire, uh, you know, uh, production environment, that question alone, um, you know, trying to answer that, uh, is, is something that I think is really important to try to tackle first. It's kind of a foundational here, here.
Yeah. W we ha we had a question here that I'd like you to you guys to, to relate to, but it also brings us to a broader topic and, and, um, you know, one, the question was to what degree are people not upgrading dependencies because they fear of breaking things, you know? Um, uh, and then basically they enable these, uh, more vulnerable components to, to pursue this. Um, and, um, and that, you know, that's, that's the, I think that's a major question and it kind of brings us to the broader topic of how much, or what can DevOps, uh, do about this, you know, is this a, is this a security issue or is it a dev ops issue? Maybe it's both, um, we know what are we who, who, who's, what, what's their role who's, who's supposed to be doing what, in this case?
I think that's a great point. And, and, and for me the question around, um, how to avoid, like staying on all dependencies is a tricky one, because the kind of underlying components that, that you use tend to have, you know, they don't have these enterprise life cycle support life cycles of five, 10 years. You look at Kubernetes as an example, it was at nine months until recently went to 12 months. And really, I think dev ops has a huge part to play because, um, in an environment where you're deploying rapidly and you're used to deploying rapidly, you're used to that fast change cycle. It becomes much easier to make smaller changes and smaller changes will have less risk. So if you can get yourself into the world where you're saying, well, I can deploy every day, every week on a short cadence, making small changes each time you're less likely to have made a hundred patches and then have to work at which of a hundred caused the problem. You may have only been five changes, and then you'd say, okay, I've got much smaller set to work with here. So I think that element of dev ops processes and making sure they're very thoroughly embedded and people are comfortable with fast regular deployments can really help software supply chain security because you can embed it with less risk.
Yeah. And I completely agree with that assessment. Um, the, the other, the other kind of thing, I think that's somewhat complicated is thinking about that, that exact topic of, you know, uh, there's, there's one choice that could be made to say we always, even if we're, um, you know, uh, deploying rapidly and making, making small changes, we're always trying to keep to the latest, um, uh, dependencies that can work. But, you know, there's some serious sort of functional, uh, you know, uh, considerations that need to be taken into account as your dependency list starts to grow. And so, you know, practically what, what we've seen a lot of organizations employee is to really decide whether or not to update a dependency based on, again, insight about what is happening with that dependency. Right? So if we, if we have a practice, uh, when, as we're developing software of inspecting our dependency lists, knowing exactly what those dependencies are and a way to actually consult, um, you know, vulnerability sources that say, what, what new vulnerabilities have come out yesterday, right? And if there was a couple of dependencies in there, knowing that information can give us exactly that type of guidance to say, well, we don't have to update all of our dependencies necessarily all at once to Ari's point. You just need to choose the ones in a targeted fashion because something has happened updated, make sure it's tested, do functional testing and move on. Right. So I think that kind of, um, uh, insight, uh, early on in, in your build process can, can really help here.
I mean, arguably dev ops should be better equipped to do this then traditional, you know, infrastructure plays and, and, and kind of waterfall SDLC, because, you know, they have these methods of, uh, Canary deployments, blue-green deployments. I mean, you, you can more easily test dependencies before you deploy them or do it gradually and see the things don't break. So I would think that actually outside the realm of dev ops things are that obviously, you know, things are a lot harder, right? I mean, when we talk about non open source non DevOps environments, where you have this, you know, database that's been running, um, for three years with no upgrades, because people are afraid to shut down the servers, the server in case it won't reboot. Um, I think those are the risky ones. And I think a lot of the, the big ticket, uh, supply chain attacks that we've seen where, of kind of older software or things that were, you know, in there for years and nobody paid attention to them. But do you agree with that assessment?
Yeah. Yeah. I would, uh, the, the, you, you see this kind of interesting, uh, clash and my, my previous life was as a pen tester. So I used to review people's systems and find out that they weren't patching or long periods of time. Um, and that absolutely, you see this kind of, and there's a slight, obviously interesting thing that you see these industries that are used to moving in that way. So they're used to the multi-year are now kind of going to cloud native stacks. And I think there's a big challenge for them to make sure that the, you know, walk the walk of dev ops to go with this move to the cloud native software, because they're going to have to, um, because they can't take the kind of traditional, you know, I put it in place and then it sits there forever until it gets replaced life. They're going to have to move the, the kind of policy. But yeah, I've seen a lot of environments were very old software are, is your big problem. And an unwillingness to reboot unwillingness, willingness to rebuild because you've got pet server or pet database or a pet impeccable NetEase cluster. Um, and you don't want to do it then that's where you start running into problems. Yeah.
And I think th the good news here also is that, you know, th this, this, this idea of being able to update incrementally and rapidly, uh, in, in almost an automated fashion, um, you know, it goes beyond just value from a security perspective, right? And so there's, there's multiple reasons why being able to keep things updated and being able to consume new code commits and bug fixes, and have a lot of automation to get that stuff into production, as fast as possible, you know, even without considering the security, uh, benefits of being able to do that, uh, there's other, uh, you know, good reasons to, to put those practices in place that organizations are looking looking for. So I think the combination of having, uh, that, that process in place gets you some, some good security of value, but it also gets you some good competitive advantage and that you can make changes and add new features and, and do things a lot faster, um, because there's multiple value, you've got the potential that more organizations will be adopting this type of technique.
And what about the security of the dev ops tool chain itself? You know, things like CNS CICB pipelines get help, repos, Docker, registries, and so forth. Do you see those as risks?
You have someone
You want to go down?
Oh, I can say yes for sure. Um, you know, those, those are places where, uh, you know, malicious actors and hackers to get in and, and, uh, any, any place that has the ability to modify, uh, the software that's ultimately being produced or, or, or deployed, um, is, is an area of risk. Right. And so I think, uh, there, um, a lot of, uh, DevOps teams are adopting techniques where change control is, uh, as being managed with, um, you know, infrastructure is code and those other kinds of techniques that make it a lot more, uh, sort of programmatic and reviewable is really important, but absolutely those developer infrastructure components are, are, are risky needed to be secured.
Yeah. Yeah. I absolutely agree. And I think that the thing is for organizations and for developers, is to realize that all of those services are part of your attack surface. Um, these are all elements that you need to consider when you're thinking, where am my controls? Um, I, you know, I've, I've seen, and, and th the challenge, I think is the fact that we're all on the internet now, you know, all of these services are, are, are publicly available and you are one access control, mistake away from being exposed. You know, again, going back to my previous life, I knew companies who developed all their software and get hub, and some of their developers would essentially clone a repository and then make a fork of it into their public hub, because they want it to work on something. And weren't ready to commit it back.
If that had been on a private system, people would never have known, but it's on get hub and you can search it. Um, so there's the element of that. And also the element that things like CIC D systems, so have actions or Jenkins are essentially free compute resources and attackers like free compute resources, because they can misuse them. So anything of that nature, anything that lets me run code is, uh, is always going to be a target. Even if it's not for super serious purposes, people will just steal your credentials so they can run up a bill for you. Uh, and you'll get a bill for whatever, when, when, when, when they had done that. So there's definitely, I think that whole area is, is one people have to quite live to because there's so many reasons for it to be attacked.
Right. And we, and we've seen this happen with, uh, free versions of, um, online tools, SAS, SAS, the idols are being abused for crypto mining in that case, but it makes you wonder, you know, if people could have used that tool for crypto money, they could probably do it for other things too. Um, so what do you guys see in terms of things that organizations are implementing out in the field? I mean, you know, what, what steps are people taking to actually, um, you know, mitigate the risk of supply chain attacks and, uh, you know, we talked about general community initiatives, but what do you see in the field?
Yeah, I think, I think in some sense it's early days. I mean, there's a lot of areas, right? We're seeing some organizations who have already invested, you know, for years, uh, in, in this type of area or they have very serious, uh, you know, policies around, you know, which Prudential's are used for what, making sure that change control is deeply managed, you know, understanding, um, you know, even to your point earlier, maybe, maybe not doing source code analysis of all of the open source dependencies coming in, but maybe a pretty good some subset of it. Right. And so there are practices in place I think, you know, for, for organizations that aren't really at that level. Um, yet, uh, more and more, what we're starting to see is, is just simply the focus on supply chain security, which is, uh, is an interesting event, right?
And that it's kind of codified a number of different security controls and things to think about, again, related to your developer infrastructure, like your CIC D tools, you know, deeper inspection of what open source things and other dependencies are coming in more consideration on the output side saying, you know, realizing we're part of a supply chain, what do we need to produce at the end of our supply, our own internal, uh, developer process so that we can provide more information to the next consumer in the chain, right? Actual focus on those pieces of the story is something we're actually seeing organizations put time and energy and investment into now, uh, which is, which is a really good thing.
Yeah, I'd echo that as I said, early days. And I think there's a slight, one thing people shouldn't feel too worried about is if there's a huge number of new terms coming at you with supply chain. So if you start hearing things like ass bomb and signing and lots and lots, don't worry if you've not got a handle on all of that. A lot of that still developing and it's early days, um, that said, I think there are some things I'm seeing with good takeout, and there are things like automated scanning tools. So just vulnerability scanning of container images, you know, it's the kind of thing that could, can be done quite easily. It's, it's not, it's not super difficult to get started and you can get a quick return on investment. You can see the results quickly. You can see the value quickly. The other one that I think is getting a lot more traction just starting now is, um, more emphasis on infrastructure as code scanning.
So, you know, we've got this great benefit in, in, in this automated cloud native world where all our infrastructure is, is described as code. Well, that opens it to static analysis. And even again, if you're not doing the super deep things, you can do point checks, basic checks on, you know, are my containers wanting to be privileged. This is a really easy check to do. Um, it's literally a string match, but you can stop a dangerous thing, get into your environment. And I think it's there that I've seen, like people picking up quite easy interest and that's good because getting experienced with point activities, getting those comfortable, working those into your environment, opens you up for the next step of saying that we've got that happy. What, what do we need to do now? Um, so it's building that process up rather than trying to big bang it and say, I'm going to fix supply chain security with one thing.
Yeah. I guess, I guess you can stop supply chain attacks, you know, tax that originate the supply chain many different in many different points along the way. Right. I mean, we want to stop it or you want to intercept it as early as possible, but if that's not possible, you have to have these multiple live peers that will prevent it from at least the at least control the damage later on. Right. Yeah. So, um, there was an interesting, uh, point raised earlier about, um, about, uh, service organizations. You know, how as if you're a service organization, like, you know, for example, the SAS provider, how do you balance between the frequent deployments and clients' expectations of not having, uh, to do frequent changes that may risk service, uh, delivery. You can also think of internal customers, you know, uh, like a service delivery team versus SRVS versus the Def in that respect.
I think for me, that one really takes into the idea that, of doing partial deployments. And you can see this with organizations. So, you know, we're not going to put a version up, but we're not giving it to everyone. We're going to do a subset of our customers. Um, and you see it even with like the big cloud providers where Google will have different channels and they'll have a look of a kind of early adopter channel for their services that are mainstream channel, and you can choose which channel you want to be, because some people will want like the absolute leading lik latest version and other people will be more conservative. So if you can, as an organization say, you know, I want to, um, I'll, I can segregate my customers, uh, and I can give them different offerings depending on what they want. That's one option, I think that can help. Um, but it is a hard choice to make because they're dead rights. Stability is, is all in SAS. Um, but, but then if you leave it too long, you'll get an notation and, and we'll see where that goes.
Yup. Yup. I agree with that. And I also think, you know, uh, w one of the important things for a, uh, a SAS provider, uh, to do is, is that can really help a lot. Um, you know, building off of Rory's comments is to be transparent about how it is your organization does this, right? Because, you know, being able to describe to your customers, we have a beta channel, we have an alpha channel, or we have a, you know, um, you know, uh, sort of new features over here and, um, you know, more stable features over there. Or we, we do, you know, this amount of time where we have an old features supported before we switch over, and that boundary is clearly communicated through a major version number or a release number, you know, that, that kind of transparency about exactly how your organization moves your product forward and your compatibility layers forward. Uh, I think I is really important and can really help get your customers onboard, uh, with what it is you're doing. And then, um, try to keep everybody moving along with you.
Um, I guess, moving to kind of more specific things that were specific initiatives that we're seeing, um, how would you guys describe, or what do you think that the chances of, uh, things like signing at the station, uh, being the way out of this? Um,
Yeah, I wouldn't say that the way out of this. Um, but, but they're definitely component, you know, everything I'm sending keeps coming up and it keeps coming up, you know, every time, every couple of years. Um, and, and, and it gets kind of when people say, well, some people over-hype it and some people will say, well, it doesn't really do much. And think there's a middle ground. The key with, with things like signing is ease of use. And that's where we've seen it fail before. You know, we've seen COVID people come up with with great prop architectures, but they can be a bit overly complex, a bit hard to get started with. And, and to be honest, the one I'm really excited about, I think is showing great signs is six door and co-sign, um, they've, you know, the basic signing process, there is super simple.
Uh, it's something that anyone can get started with. You can start sending your container images and your binaries today using six store are using co-sign, but then they've got kind of larger idea of how to build that out. Um, the on ramp there is nice and small, so I'm, I'm, I'm, I'm hopeful, you know, again, I'm kind of expressing hope here for the future. I'm seeing good signs. I'm hopeful that this time, things like that. And it's just like that, that makes, I mean, signing a simple thing to people to get started with, um, will help and people will actually get adoption this time.
Yep. And, uh, you know, it's interesting, you know, when, when I think about signing and attestations, you know, I think I'm, uh, I'm, I'm in a similar, um, uh, or have a similar perspective and that it's not the thing that's going to solve all of supply chain. Right. It's an element, uh, that, again, I, I think of as another element of assurance, right, everybody's trying to figure out how do I trust something right, that, that I'm bringing in or that I produce. And, you know, really that, that notion of trustworthiness in, in, from my perspective, at least comes down to, if you can get enough assurances of certain types and they all kind of add up, then they all sort of bolster this notion of Trump's worthiness and signing. An attestation is an example of assurances that can be added to your, your collection of assurances when you're inspecting a software artifact or an element, because it tells you something about provenance.
It tells you about something about, you know, how something you're pulling in hasn't been modified or tampered with, you know, that's, that's one type of assurance, but there are other types of assurances that you also need in order to really address the, the, the type of trust that you're looking for to get really good supply chain security. And so, um, yeah, I, I do think it's, it's really important. It can solve a lot of, um, you know, vectors where problems can happen, especially around tampering. Um, but it's, it's not the end all be all. Um, and finally, I'd like to echo what Rory was talking about. You know, the six star project is, is very compelling. And again, a lot of it has to do with how simple it is to start signing things and the sort of flexible support that that project has for, for different ways of, uh, signing and doing attestations.
Right. Okay. And then this, this point, I like to really kind of change pace a little and ask you both, what do each of your companies do to help, uh, deal with, with this supply chain risks, uh, Daniel, why don't you start?
Sure. Uh, so, you know, anchor has been in a position for, for quite a while where, you know, our, our fundamental idea technology, what it does is, uh, historically we take in inputs, uh, in the form of say container images, we'll take those container images in and do a really deep inspection kind of categorize everything that we find into, you know, known elements, like tougher packages for all of the different language ecosystems, you know, uh, Linux distro, um, you know, PI packages, windows, uh, knowledge-based article updates. We do all sorts of categorization of, of container images and provide that information as an accessible software bill of materials. That's something that incor has been doing for, um, for a long time. And so we're, we're now that that folks are starting to look more deeply into that notion of being able to get more information about what software is running in production are being produced.
That capability is something that, that anchor is able to provide, uh, to users and customers. So getting that information about software that's coming into your environment and, um, you know, being produced, uh, at the, at the, uh, the other side of your development pipeline is stuff that, that core technology can do. Uh, we've got open source tooling as well, uh, that, that play critical roles in this story. One of them's called sift. It's an S bomb generator. You can point it out a code checkout, you can point it at a container image, and it will generate a pretty rich, uh, uh, software bill of materials just by itself in a variety of, of different, you know, standard, um, you know, output formats. And we also have another tool that can take that S mom and do something security related with it, namely to produce a vulnerability scan. So it shows that relationship between, you know, the, the software bill of materials, the information that you can produce about what's there and the security, um, you know, uh, function like doing a vulnerability scan. So we're putting a lot of effort into making these tools available, both to the open source community and as products, uh, for main core. And they fit very squarely into this developer infrastructure, kind of the left-hand side of, uh, organizations to we're pulling in software building throne, and then producing high quality metadata next to their artifacts at the outside.
Cool. And worry about it. What about, uh, Aqua?
Yeah, so I think obviously we, we see this all the same problems, uh, as an quarter, and we're looking at a lot of things like container scanning and image scanning. So supplying, securing that element of it. And again, combining the open source and commercial approaches, the some other elements we, we we've, we've been focusing on a lot, one of which is trying to go beyond the scanning element to look at dynamic threat analysis. So looking at, you know, when you're building a container image, for example, saying, running this in a sandbox and looking at this behavior and saying, does it do things which appear to be malicious? So static analysis is great. I mean, it's absolutely something you need to have, but adding that element of dynamic analysis prior to thing is getting to production is something that we've been focusing on quite heavily as well.
And again, we've got open source projects. Uh, I'm like Tracy, which can help, uh, look at dynamic analysis. And then, um, uh, we've got, uh, open-source projects for scanning or, um, uh, trivia as well. And trivia is a great project for supply chain work because it covers container images, but also things like IAC. So you can get the IC element of scanning and getting best practice in there as well. The other element I think that's been quite important is, you know, you've got all this great work in the left side, you've got this great work looking at development, but when you're pushing that into production, having a gate, you know, having an admission control that stops and says, you know, has it been done the work they've done before? Not somewhere else that was really focused is on helping people have something which can apply those policies and ensure enforce them when things are going into those production environments. So it's really about trying to get that whole life cycle picture, um, and, and get different approaches because I think like anything insecurity, no one approach is going to get you all the way you have to combine them, uh, and then to try and build up as best the picture you can have your security.
Great. And I think, you know, we're getting quite close to the end of our session. So I wanted each of you to give a few 1, 2, 3 points that you think are the key things that organizations should do now. Like what, what, you know, tomorrow morning, what, what should they embark on to protect themselves against supply chain risks?
I, you know, I, I should pick up something Daniel said, which is the first thing I would do is try and work at what you've got and that that's not, you might find it's harder than you think it's going to be look at what are the libraries you're depending on what are the SAS services you're depending on, you know, trying to get a picture of all of those. And then, you know, you can start, as I've mentioned with, with point activities. So, you know, don't think I have try and solve this today. You won't, but look at something like getting an source, vulnerability scanner, you know, start with the source projects. They're, they're easy to get started within the free is always a good place to start, um, and scan your container images, start scanning your infrastructures code templates, and just look at the high-risk issues. You know, again, don't worry about getting everything done today, just start at the high risk and start working your way down. Uh, and that's where I think you'll build familiarity and kind of like, you know, then as you move on, you might say, well, I need something more complex, something more powerful, but you can start there, but jobs, zero work at what you've got.
Yep. I think, uh, you know, from, from, uh, from, uh, where what to do tomorrow morning, uh, you know, I, I absolutely agree. And I also think it's important to sort of really step back and look at your, your software chain, right? Your build process and start thinking of your, your, that chain right there, or that, that process as a link in, in a global supply chain. Right. So you've got, you know, I'm, I must offer producers and I have inputs. I have processing in the middle, in my CIC D and I've got an output, right. And that is a link in a more general supply chain and start thinking about what are the things I can do on the input side, what kind of processing can I do? And what should I produce on the output side? Right. And if, if I do that, and everybody starts to see that I can do that.
Other people do that, and then start referencing back to some of those, those, uh, those process initiatives, uh, coming out of the CNCF and the open SSF. Those are, you know, you'll find, you know, detailing general models for considering all software development in these terms, and then how to map some tooling in and start to generate the kind of information that if everybody generated we'd be in a better spot. Right. And so that just sort of, as a model, I think is important to understand, because a lot of us here are in the position where we're designing these systems. So taking into account where on the input side, am I going to need to generate information? What kind of processing, um, do I need to do to generate these assurances that we were talking about and store them in such a way that I can include them in my outputs, right. That's the kind of, uh, you know, general practice that I think is going to really substantially assist in, in elevating, in protecting against the supply chain attacks generally.
And Daniel, do you think that, um, companies should be a bit more restrictive on their inputs, meaning, you know, kind of stop that free for all of, you know, grabbing open source packages from wherever and using them, um, to build things quickly, but not necessarily safely?
Well, I can't say generally, because I think it actually really matters, uh, what, what it is you're doing with your software and what the consumer of your software is expecting. But I do think that one of the themes throughout this entire discussion is understanding what your inputs are, is probably the most important thing, because you can't make decisions about whether or not you need to be more restrictive or what kind of policies you want to assert over your inputs, without understanding, you know, what the, what the functional requirements have pulled the, you know, made it the case that you have those inputs in the first place without understanding what they are.
Yeah. I would add one thing to that is just so that I think that there is a I'm illustrate right there. Um, I think that there is an element of, um, of treating like any dependence that you have as like a cost. So there's an element of every dependency I add here is another thing I have to scan. Every SAS service I add here is another link in the chain, right? And the longer my chain is the more points there are for someone to attack. So I think that there's that element of, I think you're right. You can't generally say, well, just cut down. But you can say to the critical angle, you know, there is a cost to what I'm doing. Uh, and the cost I wouldn't recognize immediately. I will only recognize it when one of these links breaks and suddenly I'm in a, I'm in a bad place.
I'm a longer chain. I've got the worst. That is, I mean, one of the things I always say with container security, when, when people say to me, you know, how do I make my vulnerability scanning easy for my containers? My answer is, see if you can install your packages because a lot of container images will bundle things they don't necessarily need. They're just there because they're dependency or developer dependency. And there's an element of saying, if you can reduce that, it makes your job easier because you got fewer things to focus on. So it's worth considering, but yeah, if things are there for a reason they're needed, but maybe there's some things that could get reduced. We lost
Down there again, it looks like our moderator lost connection with
We've lost our moderator. Um, let's see. I think the only other thing that I think that, that I would say about this in terms of where to go next is, um, is a kind of a watching brief. I, as I said, I see one of the things I've been looking more into exactly where people are going with salt with supply chain recently, and there was a lot of activity. No, there's a lot of activity around things like or in things like signing, don't feel that, you know, you have to be the masters of that. Now I think that it's fair to say that some of these are still developing. I mean, you look at things like the S bomb specifications, they're still moving what they consider to be part of their school and adding more security features. So it's not that everything's there now. Um, but it's something also to keep a watching brief on, um, get started, where you can, but realize that things are gonna change quite rapidly over the next couple of next months and years to come.
Yep. Yep. I agree with that entirely. And I guess as, as maybe a, you know, uh, last statement as well is, you know, I, I do also think it's important to sort of encourage everybody to, to realize to Rory's point we're, we're not gonna do, you know, overnight achieve all of the security things that you might be able to add to your environment, but that doesn't mean that, you know, adding these assurances and adding more checks, uh, isn't extremely helpful because the more of these things we add, the more layers of security we put in place, the smaller you are, the less likely you are to be a target. Right? One of the things that a lot of malicious actors will do is specifically look for, um, you know, environments and organizations that have very obvious, uh, weak spots. And, and maybe they'll come in and find that, oh, you have some, you know, basic stuff, but not much more. And that will actually incentivize them to look further, right? You become a target by not having these things. And so by very transparently and publicly, and, you know, obviously having security controls and threat models in place, you're, you're putting yourself into a category of, you know, um, projects that, that are paying attention to security, and that makes it makes you less of a target, I think, than others. So I think it's important to do some of these things and to make it as explicit as you're comfortable with, um, that you are doing these things
Awesome. I think at this point, uh, we'll wrap this up. So I'd really like to thank everyone who joined us for this session and for asking, uh, very interesting questions. Uh, and of course our, uh, vendor dome participants, uh, Daniel Marmee, CTO of Encore. Thank you. And Rory McCoon from Aqua cloud native security advocate. Thank you as well. Um, I'm Ronnie, Osnat the VP of strategy at Aqua. Uh, it's been a pleasure having you. Thanks.
Uh, thank you very much. Thanks. All.
Gene Kim’s SRE Playlist
