Las Vegas 2019
DevSecOps Workshop - Security at a DevOps Speed
It's 30 times cheaper to fix a security defect in Development vs. Production, yet Security is often treated as an afterthought and a bottleneck. It doesn't have to be that way.
Join DJ Schleen for a hands-on workshop, as he shares tips and best practices for building better software, faster. Attendees will be given a chance to execute an attack against the same CVE that affected Equifax; then use Sonatype's intelligence to understand and remediate the vulnerable libraries and verify the attack no longer works.
DS
DJ Schleen
DevSecOps Advocate, Sonatype
Transcript
This is a bit of anger. So this is built as DJs talk, but really I've given this a whole bunch and I wasn't supposed to be here, but now I'm here. So I'm going to do the first part. And then when we come out of the demo, we'll bring DJ on. As we open it up to more discussion, um, he's much smarter than I am. I'm just really good at this particular presentation. All right. So with that, I like to start this with just a little reminder that this has been on the top 10 for over six years now. And I've just this year have really been coming back to this. We used to talk about this a lot, four years ago when I first joined Sona type. Um, and so, and just so you know, before that, I was at a couple of fortune 100 insurance companies where I was a practitioner and I was, you know, doing build and release management and starting dev ops initiative.
So, so I've walked. Um, but you know, it's been out there for a while. And so, um, so before I go any further though, how many people in the room identify as a security professional? See, I love this. You'd be adding second to DevSecOps. I feel like we get security. How many folks in the room identify as a developer? So that's good. There could be some more, and then the rest of you QA or management or some other function. Right. Alright. So I just like to remind this for me, this was a memorable day. Um, you know, so back it was June of 2013 when this came out, but it's been there for a long time. And yet in our most recent state of the DevSecOps community report where we, you know, got input from over 5,000 folks, um, we still see that 62% of the organizations out there don't even have an open source governance policy.
And so this year I'm just going to ask everybody I meet Y you know, do you have one? And if not, why do you not have one? This is my mission this year. And it, and I should highlight that. Um, although in 19, I have data in here from the 18 and the 19 report. So if you think you've caught me, it's just cause there's two different numbers in 19, it's actually 24%. Self-identified said they were, uh, thought they were breached as a known, uh, open-source component, which was up 70% from 2014. We're only about 14% of the people identified that they thought they had been breached, uh, for the same thing. And, and then the last bit at the beginning though, is that your apps are made up of 80% of these things. So this is probably if you're a security professional in the room, this might be a bit of a new bit of information to you that the apps your developers are producing are made up majority of components. They borrowed, right? We think about the code they wrote. We think about static code analysis. We think, you know, dynamic testing is going to go test that whole app, but we aren't obviously taking the time to simply identify what's in our apps and have the information at our fingertips. And what we know about those components to address that a nine, right, the eight nine says if it's known vulnerable, stop using it.
But what we do know, thanks to this conference, which I've had the pleasure of attending every year, since it began down at the airport where there was 300 of us in year, one went back in 2014, I think 13 or 14 was the first year. Um, but since that time, we know from our report that, uh, 47% of the respondents to that survey are now at a point where they're deploying multiple times a week. So that's really exciting. And if you anyone, so who in the room thinks they can deploy multiple times a week. So it's not 47% of this off, uh, this group, but we have some folks in here. So that's good news, good news for you is you're moving fast enough to have a chance in the problem we're about to present. If you are not, if you did not raise your hand, you're going to need to speed up just to be begin to address this problem properly.
So back to those numbers of open source governance, um, this drives me. So we had folks identify themselves if they thought they were a mature practice or not. So again, self identified, but I found it interesting that only 62% of the self-identified mature practices have an open source governance policy. So my retort is you're not as mature as you think you are. If you don't have this one basic concept in there, but they're probably doing, you know, there's, there's a hundred things we need to do. And they're probably doing, you know, 95 of them. So they, they think of themselves as mature. Um, and so this is that 24% number that doesn't jive with the other slide, but we know that the self-reported, um, attacks on this vector has been going up and it comes as no surprise. If you consider that over half the organizations out there, aren't even looking at this information, which means it's flowing into your apps and it's out there in production facing the web.
And there's all the, you know, the thing I like to point about out about the nine is it encompasses several of the other attack vectors of the other top 10. So you can inherit cross-site scripting, you can inherit remote code execution. You can inherit other oys vulnerabilities through that one sub category, right? So these often lead to really bad things. The other thing that co coincides with us, so not only we, you know, we have this big problem, we're not proactively doing anything about it for the most part, despite it being six years old. Um, what I like to point out is the bad guys are doing dev ops better than we are, which is to say the minute there's a responsible disclosure on a security vulnerability, they're in a position to turn around an exploit and attack you within three days. If so many of you do not raise your hands that you could release every three days they can write.
And in fact, the NSA's reported. They often see stuff within 24 hours. So they are turning around and weaponizing these things much faster, um, than, than we can get our own software out. So that's the tension I'm talking about. We're going to have to get to where we can automate faster than them, or at least as fast to keep up. So this is a big change of this is where we we've got three days from the time it's been announced to the time you could effectively be attacked, combine that with the fact that, and so again, 18 numbers were 15,000, but at 19 it went up to 21,000 new releases every day. So every day there's 21,000 new open source components available for your developers to use. That's just an overwhelming amount, right? There's just, you can't even begin to wrap your head around it.
And even if you started one day, you would quickly be overwhelmed, certainly by lunch the next day, it's just too many too fast, but we don't care about all of them. So here's some stats. This is something I just played around with recently over the weekend, we know that the average organization consumes over 300,000 open-source components a year. We know that because we're the stewards of Maven central. And so we see Maven Central's download reports, but we also talked to the folks at MPM new get and the other package systems. And so if you go to like module accounts or someone, you can see that this is roughly, um, or module counts as of the new stuff. But, you know, so when we aggregate all that data in the state of the software supply chain report, we come up with an average number of 313,000 of which most companies are using about 8,200 component versions, which means it's, uh, you know, you might, and which comes down to 2,778 individual components.
So you're looking at an average of about three versions per component. Now these are all averages. So there lies, nobody's actually doing this and most people have lot more versions and you know, it skews and other dimensions. But what I thought about with this, and so that means in theory, you should be vetting. If you had some open source governance policy, you would need to vet 8,200 components a year on average. So let's do some math who here has an app sec team of 15. Did your hand go up or did it? There's not a single app, sec team of 15 people in here, right? Who thinks they're on the biggest app sec team in here? Who's got a team of five, got a team of five, about three, you got 10 or so team a 10. That's the biggest team here. How many of you support 10,000 developers?
What do you support? 5,000 developers. You have a team of 10 supporting 200 developers. If these guys are hiring and you're an AppSec professional, you want to talk to this guy. That's awesome. I've never heard that ratio. I usually get about a thousand to one at one app set guy for about every thousand developers is usually what I'm hearing. But so if you don't have a team of 15, you can't even possibly begin to manually just process the, the flow of open-source components coming into your organization. You just it's physically impossible. So this is why we want to turn to automation.
Um, so this was a blog post from 2017 from Susan Moore. And I'm pretty sure it came out in anger because I think it's hyperbolic. I don't think it's actually accurate, but I think the anger was given some of the things I just told you, given the sheer volume and variety of components, given the rise that we're seeing in a tax coming through this vector, given the fact that most organizations today still are not addressing this concern, right? Don't have formal policies in place and with even a manual process, right? So that leads someone like Susan Moore to in anger say, you know, by 2020, um, 99% of all, uh, exploit vulnerable exploits will be known by security at the time that we'll have the knowledge of what the attack vector could be only to watch it happen because we're incapable of stopping it where we have it, or we're unable to act fast enough. I think it's hyperbolic, but, but it's interesting. It's interesting to think. So it's just a pessimistic view that says, given all the things I just described, this would seem like a likely future because unless we change, this is what's going to happen.
All right. So now I'm going to dive into the hands-on part that I'm going to do, but I just want to talk about, it's not a totally, a lot of demos and stuff are very contrived, right? They're very sort of packaged for this one. What we did is shortly after the Equifax breach in March of 17, we went and found the rush, the rest showcase project, and we forked it off and we just took that little sample project and we threw it into a Tomcat container. But at the end of the day, that's a lot of what, like your developers might be doing, right. They're just sorta taking where something like struts or something, make reference example, maybe some project they previously worked on starting with that as a base building around it, deploying it out via Tom cat or what have you. Um, so it is sort of real world. And so even though it's an example project, it's very real world in the sense that this is the exact same way. Many of your developers might be bringing your apps into production.
So for this is the link, uh, the stretch to RCA and the Sona type nexus community on GitHub, this whole project in there. I don't expect anyone to be able to kind of keep score here in the classroom, but if you want to, uh, at a later time revisit this, uh, you're welcome to, and I'll have an email address at the end. I can send people the deck and, uh, some information if that would help be helpful. All right. So this might be a bit of an eye chart. Eclipse is not great about changing fonts in my resolution. Let me see if I can, uh, I'll just live with it. Um, so this is the Sona type, um, IQ server being integrated into an IDE server. And the reason we do this is to address this problem, speak directly and direct it and solve it in a way that's, uh, putting this information right in the hands of the people that we think are best positioned to help us solve this problem, which is the developers at the time of coding.
Right? So when we talk about shift left, and we talk about all that stuff, this is the moment that we have the opportunity to get it right the first time and not send a defect downstream, right? The lean concept, like just do it right. Jazz. Humble. Talked about it. Gene Kim talked about it in the Phoenix project. Uh, it's an Edward Deming con concept, right? So like, how do we just get this right? And the thing is, we felt our theory, Sona type was the reason people continue to use known vulnerable components is they're simply unaware. The developers are, have these components, but there's nobody giving them the information they need. If you have some manual ticketing process, they quickly stopped doing that because it often takes weeks to get that approved. Right. But that doesn't fit with our sprints. So you can't possibly like submit for a component and get it out in the same print in the same sprint.
So what do you do? You start stop, you stop submitting the ticket. You grab the component. Anyway, figure it's easier to get forgiveness than it is to get permission and you move on. But we've, we fundamentally believe that if we put this information in their hands at this moment, that we have an opportunity to deal with this. Um, so this is, this is the app and the struts core, uh, app that's in here is going to be our vulnerable component. And this is the kind of information we're presenting, where we can see every other version available to us. And, uh, the color bars underneath it represent whether or not there's a policy violation. So we've cut, we've codified our policies as a set of rules and these, and then soda type provides all the data. And so this thing makes the request off to our server and it passes all this stuff back.
Um, and we get, we get some good visibility into this and the way we teach my entire developer training classes find a green bar with no red under it. You've all just been trained. It's really, really easy, but it you've want to find a big green bar. That means that's an aversion range that is probably API compatible. And if there's not at least a red, you know, in this case, there's orange, but there's no red. So it's orange is better than red. Um, you know, the idea with risk is not to get it to zero, but to get it down right. And an end to know your risk to quantify it, the two things right. One is to sort of suppress it. And the other one is just to, to quantify it. So let's, let's take this project in the command line and I've, I sort of had to already build and package this stuff.
So what's up control shift. I'd made it bigger already. Can people see that? Okay. I was gonna say, I thought I made a pretty big, um, so this is a pretty simple Maven clean, right? Um, it has this rational application. It has a governance tool in the newer version. So if I don't have to skip, it'll fail because of licensing. So I've added that switch to keep, keep that out of here. This talk is not about that. So we just try to avoid it. Uh, this should just go through really fast. Am I not on your wifi? What am I on? I'm on at and T oh, this is where are you? Is your zone. They went through nice, even though I'm on this eight, just free at and T wifi. That's not at all sketchy. I have my VPN VPN on at least. All right. So you can build it. And then really what you want to do is you want to come back and you want to build it. And actually in this case, I want to build, right. I build it an image called Hackney.
This might've been a bad idea. Is your hotspot. No, this is all right. And now they're going okay. Well, I already had everything cashed, so it just kind of had to get through it. All right. And then what we want to do is we want to run this. All right. So I'm going to run it. And we'll just sort of confirm that real quick. So we see mine, a new container showed up here. This is my app coming up. That's a, not one. You can't see softly tight. Let's see if I can get this to
All right. So here's the app. It's up, it's got three orders and this is a sample, uh, rest application. Uh, and the attack vector actually wants to come through. Uh, it comes through these and we'll take a closer look at that data in a minute, but this is exactly the URL we're going to attack. We could have dunk tent on any order, but we just chose order three. So to do that, now that it's running, I have an exploit that's in that, um, get hub repository. It's a Python package. And so here's what we're going to do. We're going to run this exploit, Python, all the codes in there for you to look at it. And it's also sort of articulated in our data that we'll see in a little bit, but it's going to allow us to execute a command in this container. Right?
So if it wasn't a container, it'd be in your VM and your bare metal wherever, but just want to demonstrate that we can do this, right? So it told me I'm in the Tomcat folder, right? So I just proved a remote code execution in this, right. This is out there. And it's known though, but, and also at this moment, like to point out, you know, it doesn't take a nation state to do this. I think in our minds, a lot of us think hacking is really hard, right? You have to be really smart, but the truth of the matter is the teenager down the street. If he's invited into the right dark web can download these, these attack kits and start to weaponize, right? You don't really have to be this. I don't know how to do this and I'm doing it right. So what other commands could we run now that we're in here? What about the app? Sec folks, you guys probably think better than the developers in terms of what can we can run? What, what do I want to do now that I'm on your machine and map? Find out what's on the network and then pivot into it. Yeah. There's a lot of great suggestions I'm going to at least start with. I mean, the basic thing is a numerate, right? I want to find out who I am, what can I do? What do, what do, what have you given me so far?
Okay. I D I typed it wrong. I
How's that look. So this is a by-product of the Tomcat container. If you do nothing to the Tomcat container, it's running as root D. I'm not sure if devs in here know that. I don't know if AppSec people in here know that ideally there's tools out there that would identify this when you run out. I think of Twistlock and Aqua sec as being very good, sorta runtime detectors is a sort of thing they would have pointed out to you. Uh, but so this is not good. So from here, someone like DJ would just be salivating because now he knows all the ways to enumerate and all the ways to start to poke around your system, see who, what this thing's talking to. Who's talking to it. Where's the data, or maybe I'm more interested in just mining cryptocurrency. Maybe I just want to get another payload down and start mining. Uh, there was a light coin conference here yesterday. So I'm going to go with light coin. Um, right. So this, this is the whole problem, right? Like this is just too easy and it comes out within a matter of days of this being known, all right, we're all on the same page with that people
As well, take these vulnerabilities and they convert them and they start looking at what, what else this is running on. So if this is running inside of a Kubernetes cluster and as an old cluster, and you have some vulnerabilities, now I have written a container. Now I can skip into the base underlying host. And then from there, it can pivot into the rest of your network. So it's really serious. Like one of these vulnerabilities can be chained into other ones. Um, so this is where it's really.
No, absolutely. So I just counted the password file. Another target, a big tree. I got to really stopped doing that. Another big target would be look for keys, right? Amazon keys. That would, that's another high profile target. So now, now, now that I have this access, now that I have this, obviously we're limited only to our imagination. No, that's on my system now
That's the shadow. Yeah. You got to do that. If you do the shadow, you'll get all the hashes for the password. You run them against a rainbow table. And this one I have everyone's password that see shadow. Does that look right? Yep.
Right. So there you go. See, I just learned some,
Yeah. If there's any users in here, you'd see a hash of their password. So that's
All right. So that's the nature of the problem upfront. Let's go ahead. And, um, I'm going to kill this container for now.
Oh, by the way, you got a really bad root password and that if any
Yeah. I'm sure. In the container. Yeah. Um,
So now I want to dig deeper into this data. So here, you know, like I said, as a developer, we were on 2, 5, 10, that's the version that's in here. Um, I want to get up to probably at least 17 or higher. Right. And so just in here, just as a way of, by way of example, I just want to show, if I go in here and just edit this to 20, and then this, I don't know how many folks know Maven, but stretch projects if set up properly and Maven work like this, where I changed the version in one place and it affects every stretch component. So this version is actually calling out to the parent, which then has all the dependencies, right? Cause you don't want it stretched. There's a lot of components in there and you don't want to update the version of everyone.
If you're, if you're set up that way, what happens is they start updating the versions of some and not others. And now you've got another set of problems on your hand, but if I just update this to 20, which was what the recommendation was here, right. It will quickly reask. It'll basically update the dependencies, load them on the class path. We'll rescan them and reevaluate. See, unfortunately, so when I call my IQ server locally, it has to go hit our data services in the cloud, right? So we're not like a giant on-prem database that you have to maintain. This thing is constantly being updated in the cloud. Uh, you know, it's just, there's a whole automatic vulnerability detection system that we have in front of our dedicated, uh, 70% security research team. All right. So stretch core is the offending thing here.
And if we go dig into, I mean, actually the vulnerability is and pull it up. I believe it is this one. So here's the kind of write-up we provide, uh, that's behind that data that produced that red or orange square, right? So we at the right there at the front line, we want to dumb it down to something, visual, something that's easily, easily actionable, something that you can just add a glance, know what you need to do. I'm just looking for a new version. Um, but in here we give you the deep details of it. So there's the write-up that would have existed in the national vulnerability database. And then the next explanation comes from our dedicated security team, right? And so they have approached this very differently. We've, we've tasked them to say, look, as you're doing this research and you're writing it up, I want you to write it up in a way that's really understandable by both developers, but also has enough information that an application security, professional, uh, can really be a part of the process, right? Because not, and there are two worlds don't always overlap well. So it was, it's a bit of a challenge to come up with a write-up that makes sense for both.
Now you get down into our recommendation, we talk about ways and work arounds, EV even when there is no non vulnerable version, we usually provide some kind of workaround or something that you can do if such information is out there. And we get all of that from in this case, uh, it looks like there's just an override of it or, um, of, of the class, which is typical. But in here, there was, we also shared where we did our research, right? So we shared where we got that attack package from that exploit PI probably came off of that link. And then this is where we got this information. We scored out with the attack vectors on the bottoms. Um, our guys actually always rescore it. Uh, so we look at even if the national vulnerability database scored it, we will actually run it through the vectors ourselves and think with that deeper understanding, um, and maybe rescore it.
And very, oftentimes we come out of a higher, we think it's asking actually riskier than was originally reported. Sometimes it comes out lower, but so it just depends. And, and, um, but you know what we're getting into here as a of detail and precision that starts to let us automate and that's, what's going to drive all this, right? Without this precision, if we automate against data, that's sort of mostly right. You get what you would expect out of it, right. Garbage in, garbage out, and you don't, and then you get into alert, fatigue. You don't want your scanning tool throwing tons of alerts. Most of which don't mean anything. And I understand that from the application security community, you're sort of used to that, right. You're sort of used to static code analysis. You're making, trade-offs like, how many rules do I run a run, but which gives me the depth, but it's also going to produce more false positives, but it's also going to take longer to run.
So do I do less rules not go as deep, but maybe I miss some things, right? So this is a trade-off that I think the application security professional has learned to live with, uh, for a long time, but it doesn't have to apply in this world because we know the components are components that are hashed. We know if it's the exact same component, we've done the class level analysis of this, right? And so now when we see that component and it matches our hashes and our advanced fingerprinting, we can, uh, confidently alert you to whether or not it has a problem and feed that information to the developer and use it in your deployment pipeline, uh, to get better, you know, to start to control it.
So that's the data behind it that drives all this. All right. So I made one change, one version, but it's a little, it's a little convenient cause it's stretched. So it has such a profound impact on the whole project. Uh, but if you're using stretch, you likely see the same things. We're down to three now. Uh, there is no good version of the rest API, but if I wasn't go dig into that data, maybe there's a workaround. Maybe there's something there that we can deal with. Um, Jackson data bind only, just recently put out their first non vulnerable release.
Yeah. That's been the bane that's
Been on back and forth for quite a while, since,
Although there's still some confusion on our data of whether or not it's shown up, but they finally took our advice and changed the default typing and input stuff. So they've avoided their, uh, their DCR realization issue. But so with that one change, we took a ton of risk off the table, got rid of a bunch of components because those struts components relied on other components. So things like Xtreme are gone and other things are gone. So we're down to just a few. But the important thing is that one in particular is gone. And so now if I go run that, and instead of hack me, I have a fixed one, right? So I rebuilt it with the new version. We containerized it. We don't need to watch all that, let it come up running.
One of the things too is when you look at open-source software scanning, it's really fast, right? And with those statistics that we saw earlier about the 2080, you know, we've seen 3% to 15% to 20% of the software that is scanned by open source software components or any kind of DevSecOps scanners in the pipeline, um, is code that you write. So a lot of folks actually set up a static analysis to scan all the open source components. And we'd be sitting here for the rest of the week, probably wait for that to happen. So that's not moving at the speed of dev ops. That's really jamming the wrong tool and for the wrong job. So when we're talking about dev sec ops, and we see how fast this can get information back, you know, that's, that's going to be moving left.
So just to prove we fixed it and go head rerun the attack. And I get HTML back this time, I haven't popped a shell. I just really what it's supposed to do in return HTML. So the tax gone, right? So we've proved that it was there. We proved it was real and easy to get at. And we re we proved that just by changing our version number and rebuilding the problem goes away. So that's that. So now let's
Go and we automate that patching
That we're working on it, you know, that
I see. That would be, yeah. That's I asked that question specifically, because if you can have a zero touch upgrade and if you're deploying properly, then you can always roll back sort of scary. But so it putting production code out there and getting another vulnerability like in the next five minutes, it comes up as a zero day. So upgrade now upgrade later, it's really risk to the organization, right?
So as soon as we like to talk about, and this is the thing it's Edwards Deming concept, we all have supply chains. It's no different than the physical world. It's no different than the work that Deming's brought to Japan, Toyota, Mitsubishi, and others that help them, uh, as, uh, know, really they took that work culture and it became a national national culture that helped propel their economy to be the second biggest economy on the planet rebuilding after the war. Right? So it took a while, but they did using these concepts of building quality in, and the irony there is w you know, we often think that if we take the time to write the test, if we take the time to do the extra things, we'll go slower, right? Because we're doing more things. But the reality is it increases productivity. You actually end up going faster because you take rework out of the system, right?
If I fix this and I pass it down and your security folks do a scan and there's no issue, then there's no ticket. Which means there's nothing for that developer to go pull off the queue to work on. And instead they're still writing tests, writing features or doing experiments and trying to drive innovation, right? So the way your supply chain works is no different than anyone else. Right? So things come in through your front door, we have a product called firewall. I'm not here to pitch products so much, but there is a way you want to inspect these as they're coming in. Right. And if they're not good right then and there stop them. And that can be really important because if your developer starts their experiment on a module and starts really building out a whole proof of concept or some minimal viable product only to learn at the end of that cycle, that what they were using, wasn't acceptable. And maybe it's not a simple version. Maybe they really want to change frameworks entirely or whatever that was. That could be a lot of work, right? We can, we can end up churning a lot even
Components that are coming in. You can tell that they're coming in. So for a security organization, knowing is half the battle, right? And if you can stop this at the gates and find out what's coming in, that starts getting you closer to knowing.
So a tool like this can save you 15 person, years of work. Right? And I did that math based on four hours of research, our debt, our 70 guys, that's our holy grail. If we could get two issues per researcher per day, four hours, we think we've won. That's our we're on a path of continuous improvement with that goal in mind, it often takes us some, it's usually between four and 12 hours, and this is all they do. This is all they do. These, this is their job every day. So there is as good and efficient at it as they can be given the technology that we've empowered with, empowered them with. But so this is right now, if you, so, does anyone have a open source governance policy and a manual approval? You do have some, right. So that's a lot of hours that you have to invest just on that process.
This takes all those hours off now, but now what's my next problem. Okay. Yeah. I bought my milk. It was good. I put it in the fridge now. What's my problem. It's going to great. It's going to get old. Right? It's going to go bad. Right? Software ages, like milk. It's not going to stay good. So we effectively have to ask ourselves every day about those 8,200 components. Isn't still good. Right? So we want to do that and to do that our life cycle platform, just as baked into the, you know, the CEI process and the delivery process. So as this thing's moving, it's always getting scanned. It gets the data of the moment, right? Cause our cloud thing is being, you know, since I started this talk, my hosted data service probably has 25 new vulnerabilities in it or something. Right. Cause they're just kind of constantly flowing into it.
So you get the data of the moment. Um, you can use this as an opportunity to automatically break a deployment process. We don't really recommend breaking builds. That just makes people angry. Um, but, but you know, not getting beyond QA that's acceptable, right? It can be at an exit criteria and entrance criteria. That's the appropriate place. Um, our repository product, which has just stores all that. And so from here on, I, I want to just open it up to questions about this problem. You know, if a lot of folks aren't doing it, uh, anyone has any questions now you can, but I want to dive a little deeper into how we curate this data. Cause that was the real challenge. It's one of the reasons this became an issue is getting this kind of data. Isn't easy. It doesn't just readily exist out there. Right?
We had to build a team and become a data feed. We couldn't buy a feed. You know that there, none of them were good enough to end the national vulnerability database is woefully inadequate. Let me see if I think I have some stats on the database. So here's, here's what we do. Here's our view of it. And the way I like to simplify this, if any open source developer is talking about doing open source development on the internet, we're scraping that, running it through machine learning and AI processing. And that's how we're discovering these things. And it's in their get commits. It's in their issue, trackers, it's in their dev forums, it's in all those kinds of places, but then what's the missing step. What, what they're not doing is then turning around and self-reporting that off to the national vulnerability database, right? That's an extra step that they would have to do.
And that's where this falls down. And because they're not reporting that we, we have about 1.3 million vulnerabilities that we know about that is not in the national vulnerability database. And it just gives you an idea of how disproportionate that is. It's just in the last eight months or so that the national vulnerability database added their 100000th CVE. That's a fraction, right? And not all of those are open source. A lot of those are different platforms. There's infrastructure. There's a lot of different things in there. So that's where this all falls down is that source is not reliable enough for us. And so we even did some benchmarking of our own. So we used the old wash depth check, which relies on a national vulnerability database. Um, and we did a scan of a bunch of apps and then we scan with our stuff and we set it loose with our researchers and we dug into it.
And what we found is if you're just using the national vulnerary database, you're looking at maybe getting one in six valid findings, right. And, and the reason for this, oh, I don't have the, I don't have the CPE naming. I thought I had one more slide in here. The reason for this is the way this works is on a name match system, right? The national vulnerability database maps, uh, those CVEs to CPEs, CPE rolls up to something like Apache ploy, Apache or struts. Right. Um, it just rolls up to a name like that and all vulnerabilities are associated with it, but they're not associated with the specific component that brought it in the actual class. You know, the file that had the class that has the actual offending code. So they just lack the ability to make that association. So you're looking at a lot of false positives and a lot of false negatives if you're relying on this data. So like I said, this data just doesn't exist out there. Um, so with that, are there, are there any questions about this topic or any, any things anyone wants to point out? Yes.
So let's say there's 18 application scanning, let's say one of the
Or scans or fast. So we're not concerned about the time there. It's a good question, but, but we also want to know all the apps that are using log for J right. So one of the things I didn't touch on in this was, uh, incident response, right? So let's say we did everything really well. We got our nice clean app out in prod, and now we are Dell. Now we're facing a zero day. How do you, so anyone in here now did a, the last time you got a letter from the FBI, which is probably in the last year, right. If you're a financial services or whatever, but you know, what was the last time you got a zero day and you were able to immediately report the impact to your portfolio and say, oh, we know that this component is used in these apps.
Anyone been able to do that? Yeah. See. Right. And how, how do you tack that? Do you just ask teams, do you run endpoint scanning? What do you do to, what do you do to answer that question? So the reason we do all those scanning is the simplified version is everyone's using log for J but the reality is they're all using different versions of log for J no, two of them are the same. Right. And we just need to know every app that's using every version of log for J. So when new information comes along, we understand the impact, right? So that's our meantime to detection. And then we can work with that team to fix it, ideally in the next release and ideally in less than three days. Right. Ideally for the nastiest ones, we have less than three days to get that patch into that app, rebuilt and redeployed or were at risk.
Yeah. One of the problems with components is developers will pin them, right? Like if you look for a cocoa pods, like our gem file dot lock or something like that, you're going to see a version of it and it's pinned. So it's not gonna update. It's not like you're using the latest. So there's a few places where you got to put these kind of open source scans, you got to put them on the developer machines so they can do it while they're typing or, you know, find out there. But then when you're putting it to your build server and you're starting to run that automation, you got to do it again there because by the time, you know, somebody texts it out or takes that component, the pin, the version, and then they put it through the pipeline. Maybe that's where you're pulling that in and finding out again, um, you know, that there's a vulnerability and the whole S S bomber software, bill of materials comes out of that too. So you can see what versions are in there, what components you have and the sub components too, because it's know like an iceberg, but
Yeah. How frequently do you guys release software? Is it quarterly, monthly? You do release daily. So you, you raised your hand. Oh, so you're in a good position because that's going to say like, even from my earliest cm release, frequency affects your thinking. If you only release once a quarter, it's important to rescan because it could be a month or two old since the last scan. Right. But if you're actually really, really fast, it changes how we think about it because we just did it and we just did it again. Right. So we're in a much different situation, you know, we're in a much different situation than, uh, you know, in the insurance company, we have the, uh, end of year code freeze for the once a year release for open enrollment. Right. That, that app goes out once a year. I mean, just been staged and tested and stuff, but you really only got one shot at it coming around, but it's a long cycle. So if you did all your upfront work, it's old by the time you get to September, when that starting to kick in,
I always used to suggest, um, scanning code, that's been in a repository at least on a weekly basis, so that you'd get ahead of it because you know, this is exactly the problem is you'll have code, that's checked in with pen versions and then you'll pull it out and you'll deploy it. And if you bypass any of those security controls, there might be vulnerabilities in there because of the pin version. So you keep scanning them regularly, even if they're in the repository, um, just to get that information ahead of the time.
So any other questions? So I want to make sure, and everyone had a chance to poke at this,
No talking about like GitHub or like pulling it out of your source control repository, because like a lot of people just check it in and forget it, or there's another problem too, um, where you'll have the team that has a fixed budget, they get pulled on a team, the creative product, and then they're taken off and like, nobody will touch the product even though it's deployed. Um, so that in that situation, you'll, there's nobody there to fix it. So that's another problem. But knowing that the components in there and it's vulnerable, that's going to help you sort of prioritize. And hopefully you won't have the security organization pointing back at you saying, you know, you fix it, you fix it.
Yeah, yeah. That the integration I didn't get, I didn't want it. I didn't show a ton of the platform, but the integration does in a nexus of when you have a, when you're in firewall And even without, even in our free product, you can get to, um, Oh, am I?
So there's lots of different things you can do. We set up a Docker proxy. So like anything that came down, we know about it. And when you checked it in, um, it would check in locally. So that container, or that image would be in the local image repository. So that just sort of helps get the information. And then we actually did a redirect. So if anyone called hub.docker.io, it went right to the repository. So you avoid all that.
We, in our free product, we have repository health check, which has a free cursory scan of everything in that one repository. So that's out there for free. Um, I think it's limited to public data only. Uh, but then firewall gives you that more detailed report. Now this is based on, um, the complete data set from the backend, the paid data set. Oh, actually I can't log in as that account here. That's right. That's a, I'm a fool. I switched over. I can use it on the backend. I can't use it for the UI, But this is also a scan of the repository. So it looks like an application report, but it's a repository report.
This is your last bomb, your software bill of materials. See all the versions. See what it is that
Didn't go to it. Let me try one more.
You got a question as well.
What types of scanners do you have that looks like you might be looking at the Maven file to look there. Are you going into the containers themselves?
Are you talking about language?
Well, no. W we like to scan binaries. We like to scan as is because when you scan against a manifest and here's how I like to put it. If you show me your shopping list and you tell me you want to buy romaine lettuce, there's some bad romaine lettuce out there, maybe, right? I don't know if you got it or not, until it's in your cart. I need to see which romaine lettuce you got. And even if you ask, even if you say, I want this thing at this version, doesn't mean you got a genuine copy of it. So I want to check that you have a genuine copy that, you know, and identify what you have. Um, and I want to see it in the app. I don't want it, you know, but having said that our scanning technology is a mix. Not every language ecosystem lends itself to that. So with JavaScript and stuff, we're parsing, Jason's with Python, we're parsing a requirements, text file, uh, stuff like that, right? So you have to run a PIP freeze or something. And then we re work off of that file because the binaries aren't the same ser
You're talking about the OS in general, right? Like the underlying base, the last image
Deeper. Right. but say my Docker image And the vulnerabilities of that, or it's in the,
I have three tools that I like to talk about. We're not at a red hat conference. I'm going to say anchor. If I was at a red head conference, I might say, Claire, uh, that's the OSTP level packet. And it's a very different form of interrogation. You're actually interrogating the package manager in terms of what it's brought into the OS, us for the application stack on top of it. Um, and then Twistlock, so we give the first two tools, give you a complete bill of materials. Twistlock does Dr. Work bench configuration, Aqua secs and other one, right. So twist locker, Aqua sec, I think, as being relatively similar, but they do like configuration stuff and they actually analyze the runtime. They can see if it was running as root. They can see who it's talking to. Um, and then those can all be micro segmented networks. So there's a lot of things, but those three tools in the world of containers are those three types of tools to be more fair is what I think you need to do this well, end to end to answer all the kinds of questions that we're going to keep getting
Well. Yeah. Did you another comment on that? It, you know, when you're starting to look at the containers, you're also looking at the host, right. And you're looking at the runtime and you also look at the tools. So, um, I used to be a security architect at my previous company before I joined, joined Sonatype. And when we were looking at it, we got vendor images that would come in and these would be part of our pipelines. Like we would put them into our production pipelines and we start running code through it, and they had more vulnerabilities than anything that we created. So then it comes to a problem. Like now we can scan everything, not just the code that we make, but codes that our vendors may codes code that, you know, assembles our pipelines together. And that's really important because you know, now we're bringing in a Trojan horse, really, you know, we have all these compensating controls till the cows come home, but, you know, having that internally. So yeah, you've got to use something like, uh, the Twistlock, um, anchor or Falco, which would do host detection as well. Um, from something like that, because it's still,
I don't want to stage them. You're going to have to bring them in vet them and then publish them to some, okay, this is a blessed kind of thing, right? Not that we are in favor of blessed repositories with open-source components, but you do need some way to ingest, verify and validate, and then make it available for use
The question about Docker images. And it seems like you basically answered that. Have no Rob's diarrhea images
At the moment. Yeah. Uh, th the whole Docker world, I mean, first off, we all need to stop saying Docker because doctors going away and it's getting replaced by other runtimes, other, you know, other clients, other, right. So there's K packs and, uh, pod man. And, and so we're, we're making sure that my hope is if everybody stays within the OCI standard, it should still work, but we're trying to actively test some of that. So I'm, I don't want you to pin me to it just yet, because there might be some cases where with a K packs that we can't scan yet, it doctors like St. Kleenex, right? Right. It is like Kleenex now. All right. If that's it for the questions, I know you're at a big dev ops conference now, but I would be remiss if I didn't mention all day dev ops coming up virtual online on the sixth.
Um, you're probably going to be drinking from a fire hose here for the next few days for the last thing you want is another 150 talks. But the good news about this is if you sign up, even if you're satiated here, come new year, you know, all these videos will be out there on YouTube and you can go back and revisit, and it's good content to have in your organization to share with your peers to start conversations. Right. And all these talks are much are practitioner led. So there's no vendor pitches, right? So it's all folks like yourself out there living and doing this, sharing their success, sharing their failures, uh, there's four different tracks. So I just highly recommend you've got nothing to lose, sign up, get the links, have access to the content for the rest of the year, uh, to take from here. Thank you everyone for, uh, for coming out today and putting up with the whole initial setup.