00:00:00

<silence>

00:00:12

Hello, uh, thank you all very much for joining the session. Today, we're gonna be talking about supply chain security. My name is Daniel nmi. I'm the CTO and Co-founder of ancor, and I'm joined here today by Paul Novaris, who's our Senior Solutions Architect, uh, also here at ancor. Ancor is a company that provides users open source technology, products and services that really are targeted at enabling users to bring continuous security and compliance enforcement directly into the DevOps tool chain. For the presentation today, we're gonna be focusing in on a topic that's been, uh, very prominent, uh, uh, recently in the security space, uh, which is really centered around supply chain security. First, we're gonna be discussing what supply chain in, in the context of software really looks like and why malicious actors are starting to target supply chain, uh, situations, uh, as, uh, in order to implement some of their successful attacks.

00:01:08

Um, we'll start off by that discussion, and then once, once we get through that, that, that piece of the presentation, we'll move over to have Paul then, uh, jump in with some practical recommendations and examples of how we can bring security and compliance enforcement into your DevOps tool chains today to try to prevent some of these attacks. So, uh, starting with a lot of what we've been seeing, uh, these days, uh, in terms of the, the bad headlines, right? We, we see a lot of really prominent and serious security incidents over the last several months. Um, namely, you know, some, some of the more prominent ones. SolarWinds was definitely something that that caught the attention of, of the security world. And in addition, more recently, there's another one from, from, from another, uh, company called Code Cov. And we see a lot of the, uh, these incidents using a phrase associated with what's going on, and that's supply chain risks, supply chain security, and a number of customers that are impacted by these events.

00:02:09

And we wanted to have a discussion about really, you know, exploring what, what that means and what's happening, uh, uh, in these incidents. And we can start with this kind of typical iceberg view in that what we're seeing above the surface is a lot of discussion about what happened, you know, the, the sort of top level hack and the exploits and, and things that are, are compromising, you know, customers and, and pretty large enterprise organizations associated somehow to some other element in the, so-called supply chain. And really though, if we look into what's, what, what actually started the, the whole incident, a lot of those attacks, and especially in the supply chain, uh, security world, are targeting software suppliers and or, uh, open source dependencies, something very much earlier in, in the, the, the story than where the attack actually happened, where the damage was caused.

00:03:06

And we can, we can see this by evaluating, you know, what is the supply chain and look at it from a couple of different perspectives. The first perspective is from the consumer view. That is any organization that is actually deploying production software, we might have something internal that's, that we're running in production, or it might even be internet facing, might be a website, an application, whatever that may be. At the end of the day, that's the software that's executing. And we, we, we call those the consumers. And from a consumer's perspective, they know, you know, this is my application that that's actually running in production. And in order to get that application to run, I depend on a number of different software elements that are all composed together to provide that production environment. Those can be other software suppliers, they can be open source, uh, software that comes in that all in aggregate, uh, comes together to, to form the actual application.

00:04:04

Interestingly though, from a supplier view, if we, if, if we as a software supplier look at what we have visibility into, it looks a little different and starts to expand out. And this is where we start to see a chain sort of forming. We know as the software supplier who our consumer is, we also know that as a software supplier, we ourselves are in a sense the consumer. 'cause we're bringing in other software from other software suppliers and open source projects to build our own software and then deliver that to the consumer. If we zoom out another level and say, for just an oracle, and we're not taking from any particular perspective inside the chain, but just in general, the chain or the graph starts to look something like this, which, you know, every software supplier is themselves a supplier, and there's a lot of open source elements in here, a lot of different pieces of software.

00:04:54

And, you know, this is the kind of thing where even this view is pretty, pretty oversimplified. Any sufficiently complex or sophisticated application running today, it's gonna have even more independent elements than what we can fit onto a slide. And so this is interesting from a functional perspective, and we might even look at this as a consumer or a supplier and try to derive some information from it or understand what, what's going on. But from an attacker's perspective, if we look at this from a malicious actor's perspective, anytime we see, uh, a, a a view like this where there are dependencies between various elements, and they all kind of funnel into something that we want to attack, IE the consumer, uh, application, we realize as a malicious actor, all we need to do is compromise any of these elements. And if we are able to actually get our malicious code or our attack successfully put into one of these environments, then that attack can actually flow all the way through to the consumer.

00:05:56

And that would look something like this, right? If an attacker would were to say, if I compromised this open source element or this software supplier's, uh, environment and get my malicious code in that malicious code's gonna make its way into the consumer's application, and that's where I'm gonna cause the damage. And then an interesting subtlety in the characteristic of attacks like this is often that while the software supplier or the open source project, you know, one of these elements in the graph is the actual initial place where an attack takes place. Oftentimes those organizations won't even see it because the attacker isn't actually trying to, um, you know, cause damage at that point. They want to do that in a way that's hidden or secret because the actual attack is taking place in the consumer. And for that reason, sometimes these attacks can be, uh, around and, and very hard to, to notice for, for a long period of time. And we've seen that with some of these recent, uh, incidents. So what we're gonna do is kinda zoom in on, okay, as a software supplier, we don't want to be the one <laugh>, um, to, to actually have a, a, a compromise that that causes damage to our consumers, even if they might be several steps away.

00:07:09

So in order to do that, let's kinda zoom in on what a typical software supplier in a modern environment, what their infrastructure and what their mechanisms typically look like. And it looks something like this. So we've got, as a software supplier, we're, we're creating software. We have our own application source code. That stuff lives on the left hand side of a, a typical process like this. And on the right hand side, we have the software is ready, it's built, it's executable, our customers or consumers have access to it, and they can, they can pull it down and, and run it. But in between, we've got a number of steps, um, especially as we're starting to see modern systems add more and more automation, where every time an application developer might make a, a, a new feature or a bug fix security update, that kind of thing, this whole mechanism kicks in and takes that source code, moves on to a build phase where an executable is created.

00:08:03

And then typically there's a testing phase, some staging where artifacts are signed and configurations are made. And finally, that, that element, uh, gets published. And again, from a te a malicious, uh, actor's point of view, these are all elements. And if there's a weakness in any one of these, they can sort of identify a weakness and put some malicious code into this environment, ultimately compromising that published software. And so from the, the malicious actor's perspective, it looks something like this at if any of these elements are, are weak and can be compromised, there's a number of different actual, uh, mechanisms or methods that can be used, um, to, to form a, a successful attack, which ultimately results in that supplier software being compromised, thus, the, the, the consumer being compromised.

00:08:55

Now when it comes to containers, right? Specifically when we're looking at software suppliers and, and other organizations who are starting to really leverage container technology, um, in order to facilitate this automation of taking new application code and building something deliverable and executable by the customer, um, containers are actually a very compelling way to do this, right? Uh, functionally you get a lot of power by being able to take your application code bundle, all of its elements, its dependencies, data, maybe, uh, oftentimes even, uh, a fairly full featured operating system all inside of a container image, uh, which encapsulates your application and gives it everything it needs to execute. Uh, very, very convenient and powerful mechanism to do that. But because it's not just the application code that's being shipped anymore and a container environment, this is also a potential source of security risks. And again, if we look at this from the malicious actor's perspective, we see several elements. And if there's a weakness anywhere, that's a potential, you know, avenue for getting malicious code inserted.

00:10:08

So when it comes to what those actual elements are, right, there's a couple of categories that we're gonna walk through, uh, to show some practical examples of how we can, we can protect ourselves against these types of compromises and, and protect ourselves and our customers, uh, from being victims of, of supply chain attacks. The first category generally is something that, that, um, uh, that, that we see a lot, and that's protecting our software from known software vulnerabilities, right? This is something that, that we should all be doing, is making sure that as we're building containers and as we're bringing in new operating system packages, new language ecosystem dependencies, et cetera, that all of that software is checked for known software vulnerabilities. Because if it isn't an attacker might notice that maybe it's been a while since you've updated your, your base operating system packages, and there could be known critical vulnerabilities, attacker might be able to figure out, well, the consumer's gonna have those same vulnerabilities.

00:11:09

That's an avenue. The second category is really around the, the concept of injecting malicious code into existing software. And typically that looks like malware charge and horses where either, you know, an attacker's malicious code is inserted into an existing executable inside of a container image, for example, where every time it executes that malicious code can be called, or a Trojan horse approach where an executable can be replaced by something that works the same way, but includes malicious code as well. The third general category is a little bit more subtle, but we, we see this as a, a vector of attack, uh, which is around software overrides. And these things can happen at that interface in your DevOps tool chain between having source code and that source code becoming an executable. That compilation or that build step can, can be hijacked in, in funny ways. And one of the, the examples we see, uh, attackers using called typo squatting, and that's taking advantage of a human error, where as a developer or a DevOps, uh, uh, uh, person in that role, you might accidentally misspell the name of some, some dependency like PostgreSQL, maybe I, I replaced the, or switch the Q in the L and just spelling it out in my dependencies.

00:12:31

And an attacker might actually publish a misspelled version of that dependency, such that when I go to build the misspelled version is brought in from the attacker rather than what was intended. There's a number of other types of attacks that all kind of fall into a similar category. And finally, there's this notion of credentials. This is where oftentimes during a development phase or a testing phase, a container image or the surrounding metadata might contain internal organizational credentials in order to do some testing or some internal work might be hard coded in the code or in configuration files. Sometimes they can accidentally be left inside of a container image, which then an attacker will watch to see all of the public stuff that's being, uh, published by an organization. They're constantly looking to see if credentials are left in these artifacts, and if they are, they can take those credentials and then perform some other type of attack, you know, and maybe in one of these other categories using those credentials. So I think with that, uh, we're gonna switch over to Paul, who's gonna give us, uh, a demonstration and some practical advice about how to protect, uh, ourselves as an organization that's building software, uh, from some of these types of attacks.

00:13:51

Okay, thanks Dan. Um, right now that we've seen the kind of conceptual ideas behind these types of attacks, we'll try to turn it into some practical, um, measures that we can, uh, take to not only detect these types of attacks, but prevent them in some cases, right? So we'll start right here. Keep this in mind. I'll refer back to this slide, these different categories of attacks. Um, before we go too far though, I wanna start by showing how Incor Enterprise actually does scan these container images and keep track of the components inside. So we'll talk a lot about the software bill of materials. When we scan an image with Incore Enterprise, the first thing that happens is our analyzer opens up the image and builds a list of facts about the image, right? Everything we can measure or record about the image will be recorded in a software bill of materials.

00:14:45

And that includes things like metadata, contents of the image, um, not only packages, in this case, alpine packages, we're just looking at a, a pretty standard engine X image here, but individual files in the image, the metadata about those files such as permissions, the sizes, et cetera, all the way down to things like layer by layer, how the image is constructed, right? So this is just a real quick layer history. Once we have all that, that is recorded in our catalog, and then we will evaluate the image. And when we evaluate the image, it's a very lightweight operation we can do very rapidly. So we can do it continuously over the images lifespan. Two things happen in that evaluation. One, we, we take our current view of vulnerabilities of, you know, what we know based on the vulnerability feeds we're consuming from different sources, and show a list of vulnerabilities that this particular image is affected by, right?

00:15:43

This is completely, um, objective. There's no judgment here. This is just what we know, what vulnerabilities we know of that this image is affected by. More importantly, the policy compliance is where we take not only the software bill of materials, but also that list of vulnerabilities and convert this into a judgment, uh, view of policy rules and what violations we've seen. So in this case, things like, oh, there's some docker file instructions here that we don't like, or there's some licenses that we're not permitting, right? So once we have those, we can give a, a, a pass or a fail score to an image, right? And let it decide at that point whether we want it to proceed in our pipeline or whether we'll let it deploy into production or whatnot, right? Several different decision points we can make there. What do those policy rules actually look like?

00:16:36

So going back to Dan's list of different categories, we can build rules for each of those categories and other things too. Um, and then we can measure each image against those rules. So in this case, we'll start, you know, with this first category of vulnerabilities, right? Vulnerabilities are pretty, pretty much table stakes at this point. A lot of tools do this kind of thing. If it could be a very simple thing like looking at vulnerabilities and saying, Hey, if this image has a vulnerability with a severity greater than or equal to critical, um, what kind of action would we take when that rule is triggered in this case? I've got a rule that says we will stop at that point, right? We will basically, uh, fail the image, right? There's other things we could do here, though, right? If we want to, uh, look at CBSS scores instead of severities, or if we wanna look at whether a fix has been published for this particular vulnerability, or how many days has it been since this advisory was published?

00:17:34

How many days has it been since the fix was made available, et cetera. A lot of knobs to turn on vulnerability. So we don't, we're not limited to just the severity of a particular vulnerability. Um, so we could build pretty sophisticated policies here, you know, measuring a couple of different things. Um, the second category of malware, Trojan horses, et cetera. You know, the, the easiest way to scan for these is to use an existing, um, malware scanner. Something like we integrate with clam. If we see that, uh, ClamAV find something, we'll stop the image, right? More particularly though, we can do really detailed checks for particularly known, um, malicious code. So in this case, like crypto miners are pretty hot right now. Crypto miners have a tendency to have extremely reliable fingerprints, right? So we can look for those traces and say, Hey, we see something here.

00:18:33

You know, it could be something as as obvious as a checksum of a particular binary that we could, we could zero in on. Could be things like what directory structure they use. They tend to be, again, very, uh, reliable and consistent in these things. So there are a lot of fingerprints we can track. Um, some sophisticated attackers might change some of these, but we can, you know, the more things we know about it, the more likely we are to catch them there, right? So we can layer more and more fingerprints as we find them moving into the, uh, category of, of, you know, software confusion. Um, there's a couple of things there. Image typo, squatting, right? This would be a case where you say, I want engine X and you type engine Z, and you accidentally get a compromised image. Couple of things you can do to prevent that.

00:19:27

Um, you know, the easiest thing would be to prevent people from using public repositories, like in this case doer hub, but other pub public repositories as well, and funnel them into private repositories. In this case, I've got internal harbor example.com. Um, if someone tries to pull an image from Dr. Hub, I will stop the image. If someone does not pull from this internal, uh, repository, uh, a registry, uh, we will also fail the image. Um, a couple of things to note here. You know, attackers obviously will have a hard time pushing images into an internal registry. A lot of times this will be behind a firewall and they won't have access to it anyway, or they may not know what this, uh, where this repository is. So there's a, a couple of other things to consider, though. How do you move images once you've vetted them and, and check them out into this internal registry? You know, there's a, a, a lot of different methods there. You can mirror things or you can just individually have, uh, particular images approved and put in that repository.

00:20:39

Uh, likewise, package typo squatting is a little more focused instead of entire images. Uh, individual language packages will be targeted. So something in, let's say in Python, right, as an example, um, a lot of methods to prevent this revolve around hardening the package managers. So in the Python case, it's pip, right? There's a thing called PIP sec, which aims to help people stamp down on this, right? Uh, in this case, I'm going to require any image that uses PIP to also install the pip sec, um, hardening alongside it, right? And if they don't, I can fail the image. Uh, another common attack, not just typo squatting, but what we call dependency confusion. So this is instead of putting a typo named packages into a repository, this is a method where attackers try to get you to use a different repository than you think you're using.

00:21:42

And then they'll put in a package that has the exact name of what you want in that repository, and you'll get that packet, uh, the compromise package, uh, instead of the, um, legitimate package from the trusted repository. So again, this varies from language to language. I'm using Python as an example here. But you'll see things like, uh, with pip, there's an INS index URL option that tells PIP which repository to pull from. So we can do things like, uh, forbid installation from a public repo and, uh, force it to use an internal repo. And if we see somebody using this option, this index, URL option to pull from a different repo, we can fail the image. Um, another thing we can do is look in the configuration files. Um, incor Enterprise has a, a secret scan facility, which is really just an arbitrary regular expression, uh, search engine.

00:22:38

So if we see arbitrary irregular expressions in files, we can, uh, generate a rule violation and block an image. In this case, I've put in a, a, a, uh, red regular expression for the extra index URL option in Python, in PIP configuration files. The secret scan facility will come in handy in a minute when we talk about credentials, but for here, we're just looking for, uh, not credentials, but for configuration flags. If we see these, we can fail the image. Um, couple other things here. We can look for things like if someone is piling multiple repositories in a configuration file, we can stop 'em.

00:23:22

A few other things here, the secret scans, as we talked about credentials, um, you know, ancor looks for regular or arbitrary regular expressions out of the box. We supply a bunch of these things like AWS access keys and secret keys. Also things like SSH private keys, right? So any of those that we see in a file, we can flag and take action on. So what does that look like in practice? Um, let me just go to another image that we've got here and we go to the policy compliance view. And you can see I've got a bunch of violations, right? All of these things, malware scan are found. Crypto minor, uh, traces or secret scan found extra index, URL for PIP or secret scan found, um, AWS access keys and files. All of these are causing the image to fail, right? SSH private keys, things like, um, the from directive in a Docker file using a base image from Docker hub instead of from the internal repository.

00:24:23

And in this case, I actually found two of 'em because it was a multi-stage build, uh, violations here, such as the vulnerabilities. I found a critical vulnerability in a Ruby gym. Um, again, this is all in our web ui, uh, but you might want to integrate this into your CICD pipeline and what would that look like? So in this case, I'm using Jenkins, I've got a build, I built this image, I pushed it to a repository, had an ancor analyze it, and we had a failure right here. If we look in the logs, we can see, oh, look it successfully queued it for analysis and got a fail result back. Uh, what, what does that mean in particular, right? So maybe my developers need to get some feedback. Our plugin will put right here in their workspace ancor report that gives them enumerates, all of those violations, right?

00:25:15

The same stuff we saw in the web ui. The developer doesn't need to change tools and log into another, into our web UI just to get this feedback so he can see, hey, the secret scanner found this, the malware scanner found this. And, uh, that developer has a roadmap to remediate. There's also tools for pushing things into things like Jira tickets or GitHub issues, et cetera. Uh, anything that basically has an API, um, you know, generic web hooks, we can hit and give them that, that, that feedback. Um, okay, so for a practical point of view, what else can we do for takeaways here, right? How can we, you know, what are some quick wins to get our supply chain security a little, a little, uh, more up to snuff? First thing I would say is make sure everything is centralized. All your CICD processes are in one place, and that all of your software goes through it, right?

00:26:12

Not only the software you're building, but software you're consuming from the outside world, right? Um, two, you want to do things like build images from trusted sources, right? That means things like using a small as possible base image, you know, Alpine or Red hats, uh, UBI, minimal. Um, and only adding the things to it that, that are absolutely necessary. Um, you know, a couple of other things here. Make sure your docker files are tight and you know, well written, and then comply with best practices as well. Uh, and also when you're building inventory, everything you're doing, right? So build those software, build materials using something like ancor, uh, enterprise to, uh, construct those and store them so you can refer back to them later.

00:27:00

Um, next tier. Uh, number three, automate your security testing and enforcement, right? So make sure that you're incorporating security checks at every stage of your pipeline, right? Uh, scan the image first and then you, you know, once you have that software bill of materials, you can evaluate it very frequently, uh, um, to see if things have changed. You can push that feedback to the developers sooner rather than later. Um, couple of other things. You know, just make sure that you're looking at the differences of these artifacts over time, right? So if you build up that repository of software, bills of material, you can go back if you need to do, uh, forensics and see when something bad was introduced. And then finally here, number four, deploy only trusted images into production, right? So don't just rely on the first scan you do right as you go into production. We can do a last second evaluation and use something like a Kubernetes admission controller to make sure only images that are passing and that have been through the entire process are deployed into production. Uh, okay. So that's everything really quickly. Um, I wanna thank everybody for their time. We are gonna take questions in the Slack channel here, uh, track four, um, and also we have our booth at the expo anchor channel in Slack.