00:00:12

And welcome to the vendor dome. So we're all really happy to be here. Um, I've participated in a few of these vendor dome events in the past, and, um, it's a really great opportunity to interact with some subject matter experts, uh, learn about trends in the industry, uh, and just get your questions answered. So I'm hoping we can have a really great discussion today. Uh, we have a really great panel of experts with us, um, across the panel, we have coverage, uh, really across the spectrum of concerns that arise in modern software development processes. So we've got coverage of first-party code, static analysis of code, the code that you and your developers are writing. Um, we have people here to talk about third party code, uh, that, that open source code that you bring in and forms the foundation of so many of our applications.

00:00:57

And we have expertise in infrastructure as code. So infrastructure descriptions that live in the code base that define really the operational context of your applications, uh, and then container scanning provides visibility into that end product as it's deployed. Um, so we're going to go around and do some introductions. And while we do this, please put questions and, uh, slack about any of these topics. You know, first party, third party code infrastructure is code container scanning, uh, you know, anything you're wondering about, um, you know, with regard to trends in the industry, what you can do in your environment, things you're seeing questions you have about best practices, you know, throw those in the chat. Um, those will be the main points of discussion. I have some discussion points with me, but it's much better if it's an interactive session driven by all of you. So go ahead and do that.

00:01:43

I'll start with, uh, myself I'm Steven McGill, uh, former co-founder and CEO of USDA, which you might remember from previous, uh, dev ops summits. We've been participating in these events for awhile. I really love the community here, um, used to have, was acquired by Sonatype in March. So I'm now VP of product innovation, uh, so in a tight and working on integrating those muse technologies into Sona types of products. So stay tuned for more news about that. So my background is in static analysis and that's what muse focuses on. Um, that's what I did my PhD work in. Um, and I'm particularly interested in how we can make static analysis tools, less painful and more useful to developers and make, just make them a smooth, low friction part of the involvement process. That's been my focus for a while. Um, so now I'll turn it over to Brian Fox co-founder and CTO of Sonatype to give the district.

00:02:32

Hi, I'm Brian, uh, as Steven said, co-founder and CTO at Sonatype. Um, I have a long background in open source development. I'm involved in a lot of projects at Apache, including the Maven, uh, Apache Maven, uh, Java build system. And also we, um, at Sonatype have been, uh, the stewards of the Maven central repository for a long time. Um, and so we, we've kind of had a long view on how, uh, open source dependencies have evolved and how the usage of that is evolved and grown. And more recently how, uh, some of the software supply chain attacks have been unfolding. Um, that's a, that's a topic I've been speaking about now for about four years. Um, you know, and, and I think with the unfortunate rise of solar winds and code Cove and everything else that's going on, you know, it, it, it's finally getting attention, which is both good and bad. Um, it's bad how it came about. Good that it's getting attention. Okay,

00:03:28

Great. Thank you. Um, and then we have Josh Stella CEO and co-founder of puke.

00:03:34

Hi everyone. Uh, great to be here. So, uh, yeah, my background is a little different. I've spent my career kind of in and out of the national security kinds of environments in the DC area here. Phew. At phew, we do, uh, uh, security for, for, uh, cloud deployments and cloud-based systems starting with infrastructure's code, uh, static analysis, all the way through analysis of the running environments for, uh, misconfiguration errors and so on. Um, I've spent about 30 years of my life as a software developer and software architect. Um, prior to founding feud was a principal solutions architect. I think it was the third or fourth solutions architect in the public sector practice at AWS. Uh, I may be wrong, but it really early days they're focused there on national security, DOD, et cetera. I see kind of environments and, uh, have spent the last seven years just purely working on, uh, this new software definable infrastructure we have in cloud and security concerns around it and correctness concerns around it.

00:04:46

Okay. Thank you. And then, uh, Tracy Walker, senior solution architect, a new director.

00:04:52

Hi everyone. Tracy Walker with new vector, a solution architect of about twenty-five years in software industry has a developer and operations consulting. Um, here at new vector, we're very focused on container security, uh, full life cycle containers, security, uh, both from a scanning perspective, as well as, um, running environments in Kubernetes and, um, able to, to see network traffic and identify threats in really become a security plane within your Kubernetes deployments.

00:05:26

Great. Thank you. Thank you all. So the first question I have is, um, you know, with modern supply chains, including tens or hundreds of dependencies code bases growing in size, um, infrastructure is becoming more complex containers pull in even more third-party software. Um, so it's great that we have all of you here today to speak to all of these, but with all of this going on, um, it can be hard to know where to even look for vulnerabilities, or if you flip that around, um, you know, what can you trust? Where do you start? Uh, so the question for each view is where you start, when you think about security and in your domain. And is there any root of trust, like to maybe starting with you Josh?

00:06:08

Oh, sure. Um, you know, I'm, I'm not a big believer in, uh, in, in trusting without verification. Uh, sorry. My dog is back here, shaking his head around, uh, if your hearing isn't

00:06:23

Right. Um, yeah. So I think that you really need to fully understand what is actually going to get deployed. And our focus primarily is on, uh, infrastructure as code and cloud infrastructure. You know, when you think about how the world has changed from the data center to the cloud in the data center days, the infrastructure was relatively static and you would deploy applications to it in the cloud, at least on well, uh, the applications drive the infrastructure and often in real time. So there's this constant mutation of the infrastructure environments and all the big hacks in cloud, uh, leverage misconfiguration of those resources. You know, the press will, a lot of times focus on, uh, you know, public S3 buckets. That's really an over-simplification of most of these breaches. They, they were more sophisticated than that and they often chain, uh, different misconfigurations together. So when you're talking about trust, uh, at least in the, in the cloud world, there, there is never a moment where you can really relax.

00:07:34

You have to check everything in advance, you have to dereference whatever code you're writing in terms of what it's actually going to do out there. Um, examine what that is. I mean, you can't just look at, uh, you know, components in the cloud in isolation. You have to look at them as combined with the other resources in the cloud. So if your containers, for example, are deployed into a, uh, insecure network configuration, um, you need to know that you need to know that context. And then also all the stuff you're building in cloud is typically mutable. There are some immutable kinds of, uh, resources in cloud, but most of them are mutable and there are lots of ways to mutate them. So even once you've deployed, uh, very often things like maintenance, windows and so on, uh, uh, allow for changes to be made and often changes are made. Even if you think it's kind of all going through the CI CD pipeline. And so, so I think trust to me in at least in cloud infrastructure world, the world I live in is all about, uh, knowledge it's it's about keeping knowledge, uh, fresh and current, and being aware of what's going on and not thinking you've gotten it right once. And therefore it's still right, but just constantly verifying constantly staying on top of things.

00:09:04

Great. And, um, Tracy, as you look at, you know, containers and operation and that side of the story, you know, what, what do you see in terms of trust and the vulnerability surface there?

00:09:15

Yeah, I'm kind of echoing what Josh said, the, you know, the trust, but verify, um, when we talk with customers that are doing a lot of scanning early in their development, uh, we have seen, uh, a pattern emerging of, uh, customers starting to use two scanners, right? Because, uh, different approaches to scanning and identifying vulnerabilities is actually a good thing, right. Uh, how the, how different products can do that. Um, and the ability to, to finally, you know, to identify those kinds of vulnerabilities as early as possible. And, um, especially when we're talking about trust and sources of truth and the ability to verify, um, what you're seeing and to do it in an automated fashion, um, additional trends around, well, if we're going to be more secure and we're going to add more security, then it starts to make sense to, uh, you know, let the machines help us automate a lot of that security.

00:10:12

So the ability to identify, um, uh, network traffic, maybe from the source of, you know, using the source of truth of the network traffic as itself, right? Not necessarily using other sources of truth, not that they're good or bad, but you know, actionable information, accurate and actionable information, right? You need information that is not only accurate, but it is complete. And that you can take action on that. You have confidence in the sources of truth that you're using, so that if you see something like, say a zero trust environment, um, where you see something anomalous that you can investigate that. And, uh, so the more automation that can make that possible, uh, the better.

00:10:54

And I'm Ryan, uh, when we're talking about process, um, that your, your topic that you mentioned earlier of, um, you know, supply chain attacks and, uh, you know, whether, whether you can even trust sort of the upstream sources of content when it comes to third party vote, uh, maybe you could comment and expand on that.

00:11:14

Yeah. So when, when I think about this topic, I think there are two interrelated things. The first one, hopefully people understand it now, but I keep mentioning it because I see so many companies haven't that you need to be able to understand what components are in your software, where they came from. Um, first and foremost, like that's an old message that a lot of us in dev ops and dev sec ops have been carrying now for, it feels like a decade, right? That, that should be the case. And yet, so many companies don't do that. In fact, last year's status, the software supply chain, um, you know, study that we did together, right. Said what 50% of them had a process, which was good. It's up from when we started in, in, uh, you know, several years ago from the teens. But, you know, uh, I like to say, you know, if, if, uh, you were choosing a car or getting on a plane, um, would you be satisfied if only 50% of the parts in that car were vetted by the manufacturer before the assembly line put them in there?

00:12:14

I don't think so. So we shouldn't be happy with that number in our software. So that's the first problem. The second problem kind of relates to the first, which is, um, so much of the software supply chain attacks that we're seeing in the last specifically, as it evolved to the last year has been focused on, um, your developers and your development infrastructure. You know, traditionally we saw, um, some of the early attacks, like the stress attacks, focusing on, uh, consumers of open source, because they were easy sort of common mode, failure targets, like you knew so many people were using stretch. You could effectively, you know, uh, fish across the internet for those applications. And then look to see how you could exploit them that that was the first sort of set of attacks on these, because it was just something you, you knew everybody was using.

00:13:02

The second set of attacks, a phase two was really where they were attacking the open-source publishers, where the, the malware or the misbehavior or the, uh, the bugs, or what have you were aimed at stealing the credentials to publish to the public repositories. They weren't aimed at trying to get to end users and, and mint, uh, Bitcoin, or steal their data. They were specifically trying to grab your NPM publisher Python, publisher credentials, right. That was phase two. The, the third phase that we're really in now are the attacks where you're seeing, um, uh, things like, uh, like the camera incident, where they went in through a Jenkins server and then got into the rest of the organization and stole, uh, video feeds. Right? So it wasn't about trying to ship, trying to get into the software that gets distributed to the end user. It was about using the development infrastructure to directly get into the company.

00:13:55

Right. And I, and I, I hammer that a lot in all the talks lately, because I see still so many legacy application security programs that are designed to protect, um, you know, in the Deming world, uh, you know, about making the factory better, they're designed to protect the cars, but they're not designed to protect the factory, knowing what's in the car and making the car better. Yeah, of course you should do that. That was my first point. But doing that alone, doesn't stop, you know, an intentional attack on the factory and that's where the game is really happening right now. So those are the kind of two key points that I like to really emphasize to, to, uh, people just starting to think about this,

00:14:32

You know, just to chime in on that. It, I think it's a really interesting point. And we see the same thing in, in cloud infrastructure where, you know, some, some folks out there are saying, well, if we've, if we've secured production, we're okay.

00:14:46

Not,

00:14:47

You're not, if I'm a hacker, you know, and I, I do a fair amount of, you know, sandbox hacking myself, dev is so much more attractive to me because the liberalities that are granted in dev the allowances, uh, are, are probably going to create many more opportunities for me. And particularly because there's probably lack of monitoring as compared to production. Uh, and, and you often see these breaches in the news. I mean, there was one, I won't name names on this one, but, uh, two weeks after a test environment was hacked, the CEO left. Right. So if you're, uh, because of that, uh, so, so I, I think that, uh, you know, the traditional security mindset, as you say, is protect the cars or build a fortress around prod when really, uh, dev and test are, are very attractive targets in this world.

00:15:40

Yeah. And it's especially magnified, I think, in a world where you're doing cloud native development, as you know, Josh, because development infrastructure probably has the keys to all the prod kingdoms. So, um, while you may have a lot of, you know, boundaries around prod, if you've got automation deploying it and somebody can get into that infrastructure doing the automation game over

00:16:03

Well. Absolutely. And, and also even if, even if you're really good about your credentials hygiene, which I rarely see to be honest, it's really hard to be good at that. But, um, even if you're really good at that, there, there are some attack vectors that just people don't think about from, uh, in the cloud, because they didn't really exist in the data center. We see a lot of examples, uh, of, uh, for instance, people using snapshots of disks and databases, which are effectively cloud backups, uh, to instantiate new databases in other accounts or in places, uh, outside of production that standing up new database clusters is now a way to steal data from a database. Well, in the data center who knew what backup solution you were using. I mean, a hacker, wouldn't try to go hack the backup solution to get a backup, but in cloud, they're all sitting there in S3. So that becomes a very attractive surface. Okay.

00:17:01

Yeah. So, um, we have a question here, how do you close these gaps? Uh, in-depth in the developer environment and staging and so forth without killing innovation? How do you balance

00:17:14

That sounds like our mission over the last 10 years, right? I mean, we, we've been trying to approach that in ways that allow the organization to define, you know, uh, policy in a way that can be automatically detected and evaluated against the components because you need, you need to be able to give the developers the ability to, uh, choose components without having to get approval on every single one. Because, um, as one of the comments here highlights, there's thousands of them, tens of thousands, hundreds of thousands in some organization, there's no way that you can deal with that on a manual basis. And so that's how we've been approaching it. But with, with the supply chain attacks, more recently, we've had to take different approaches, which, um, are attempting to understand the behavior of a typical project who's committing to it. What types of dependencies do they use?

00:18:02

Where are the releases from one of the releases? And so we can identify, um, potential behavioral shifts, um, that would indicate something is fishy. Um, and you know, those, those, um, algorithms were instrumental in early detection of what later became known as the dependency confusion attacks. Our system was flagging those since last summer, um, uh, before the, the release came out in February, um, precisely because they had behaviors that kind of looked, uh, malware like, um, and so we've taken that approach because we realized that, you know, when a new, when a new version of something appears in the repository, it's not sufficient to wait two, three weeks or months for the community to figure out that that was a bad release. Somebody hacked in what have you, um, by then you're exploited. So we're attempting to do this much, like a credit card company can detect a suspicious transaction while you're still standing at the register and not allow it to go through. Um, and that's how we've been trying to approach this problem.

00:19:04

Yeah. And you mentioned the supply chain research earlier. You know, one thing that we found last year with that was, um, it's, it doesn't have to be a trade off between productivity and security right there, uh, approaches to, to achieving both. There are companies that managed to achieve both, you know, day-to-day, um, and it really comes down to, you know, what are your processes? What tooling do you have in place? You know, if it, yes. If it takes you three months to go through some, you know, heavy approval process to get a new library, very important, you know, that's, that's a hard fight, but if you have automated processes around that, uh, you can still be quite agile. Well, it be, yeah,

00:19:43

Yeah. It's, it's, it's gotta be automated. And, and, you know, the way I tend to think of this is, you know, for, for many, many years, we've had development decades. We've had development tools that tell engineers and programmers when we're wrong, right. We've had compilers and interpreters and debuggers that tell us when we're wrong. Um, typically around, you know, functional concerns. Um, I think what's starting to happen now is as insecurity, was this other concern in the organization, it was typically not even programmers who were doing that. It was folks who were kind of manually going through and using tools and Sims and so on. Um, and there was a kind of big gap between those two. Well, if you bring the same kind of a feedback loop into the tool set, the developers are using for security now that, you know, you can create, uh, uh, again, very, very cloud focused.

00:20:44

So, uh, you can create a global network in five minutes by running one script. Um, well, it's important to know if that network you're building is safe, uh, and with proper automation that can be re that information can be returned to you. Um, just natively in a very developer friendly way. You know, a lot of people think that developers who aren't developers think developers are, you know, have this master plan in their head and then they just commit it to code and, and it goes, we actually try stuff until it works a lot of the time too. And if you add security to that, right, if you add that as one of the checks on what you're trying and your definition of working, then I think we can, we can bake this stuff in sooner. So I'm very optimistic about the future of security, because there's, the automation is going to really allow it to get baked in earlier and in a more fulsome way.

00:21:42

Yeah. Let me tag onto to Josh right there. We, we, uh, knew Victor. We kind of focus on, on Kubernetes for example, and you know, some of the things that we gain when we're using a microservice, you know, container orchestrator and platform like Kubernetes, uh, there's lots of benefits there, obviously with automation and the way it abstracts away the network. Um, but what we've lost is the ability to, you know, we've, we've shifted, left a lot of the security responsibility on to developers asking them to do, you know, network security policies and pod security policies and, and things. Again, you know, that they have to write as they're approaching deadlines or as they're approaching the end, the end of their sprint. And so the ability to automate that, so that developers don't have to do that so that we can use, you know, again, use the machines, you know, to use some behavioral analysis of, uh, you know, network traffic and processes that are running and collect that automatically and use that use those sources of truth to generate security policies that then can automatically be applied to other environments in the pipeline. That's the kind of automation, you know, ne necessary to create a zero trust environment, just to, if anything, just to limit all of those anomalous behaviors that you want to investigate later to identify where, you know, where did something enter? Is this a zero day? Have we ever seen this before? Why are we seeing anomalous processes or anomalous, uh, network activity, uh, you know, in our environment. So automation to be able to, you know, further quantify and evaluate, um, you know, weird things happening in the environment

00:23:23

Well, and the hackers are automated and that's something, a lot of people are not paying enough attention to, uh, you know, by the, the, the, the amount of time between putting a public IP address on the, on the, on the internet somewhere, or, you know, DNS record or whatever. And that endpoint being examined for vulnerabilities is like less than seven minutes it's it's minutes. And what that means is if you're not automated in terms of understanding what your vulnerabilities are, the first people who are going to know about it are hackers, not you. And security is a, is a knowledge wor it is not as much a vulnerability war. It is a knowledge war. It is being aware of what you have and denying that information to hackers is like 90% of the game. The last mile is the, the CV exploit or the, uh, you know, the misconfiguration breach. But, but the rest of it is finding stuff that's there. And so if you're not really putting that effort in yourself with automation to understand your own security posture, uh, you're inviting, um, you're inviting headlines. None of us want that.

00:24:42

Th that's an interesting thought that when you launch a new product, your first user will be a hacker. And I think that strikes the right tone and it's coming across to the audience. And we have a, another sort of like a collective question about, you know, whether this is even, uh, possible to tackle coming in. Seems like an impossible task. When you look at the number of dependencies, uh, many projects work with, uh, and, and the question is asking really like, well, the, um, while the tools are getting better, it seems like there's still like in between, between the tools and the problem, and you don't, I guess another way to phrase it is, you know, is, can you, can you be okay, but sort of a single approach, a single tool, or do you have to put multiple things in place, look across the latest, the stack, have that defense in depth and, um, what are you trading off if you don't do that?

00:25:33

Yeah, I think you have, I mean, yeah, no single tool, um, is usually adequate because the people who you're working against are not using any single tool. Right. They're using anything they can, whether it's, uh, you know, creating an exploit in, in some of the sources, you know, open source components or tooling that you're using all the way to just probing what you have that's publicly facing. Right. So there's, um, going back to what, you know, we kind of keep circling back to what do you know, it's a knowledge worker could not agree more with that, uh, Josh, that, uh, yeah, it's, it's what you know, and what, you know, versus what they know. And especially when it comes to like, uh, you know, there's a lot of CVS that don't get patched and knowing that you have those exploits, even though there may be limited things you can do with them, that's that's knowledge you should have.

00:26:24

Um, and probably the most important thing you can do to fight against the unknown is make sure you understand the known, right, the known behaviors of your applications, all the network, connections, all the processes, so that you can establish a zero trust perimeter. And not always, you know, in the old a zero trust meant segmentation and it always had to be blocking and things like that. Um, but now what we're seeing is in these environments, people sometimes just need the information, right? They need the accurate, actionable information doesn't necessarily mean you have to block everything, but to become aware of any, any activity that's happening, that's anomalous that's information you want, because those are the clues you're going to find and follow to identify what's happening in your environment, that you didn't want to be happening.

00:27:15

I'm going to agree with all that and add to it, you know, done, done well, you don't really end up. Um, so the goal in building software right, is to create business value for whatever the organization is. Um, you know, when I worked in national security, it was, it was mission, uh, as a private company, you know, as a, as a, as a commercial company, it's, it's, uh, you know, providing value to our customers, the goal of writing software isn't to make secure software it's to make useful software and security needs to be incorporated such that, uh, the usefulness is, is, is, is manifested in a safe way, et cetera. So this is always seen as a bit of a tax to do security. Well, my argument is that that doing it well is actually doesn't have nearly as much of a tax as doing it poorly that doing it poorly and doing it manually is awful.

00:28:17

Um, you know, I don't, I don't know how many times I had to go and run an application that I was the architect on through a NIST 800, uh, certification and accreditation process. And it was just painful and security theater. That's not how you can do this well, but if you automate it and you give useful tools, then, then you can actually still go fast, still leverage the, the, the wonderful benefits of things like cloud computing to really innovate quickly and compete in your market, whatever that is or serve your, your customer, um, without being as exposed. I think the other thing I'd like to point out is I, I hear in a lot of kind of traditional security, uh, talks that the, the, the risk management aspect, and this is a very popular idea. Of course, it has been for decades in security that really what we're talking about is how much risk are we willing to take.

00:29:15

And, um, you know, uh, trying to analyze that, and that's going to tell us how much effort to put in, um, one of the things that's going on now in the industry is the blast radius is getting larger in these attacks in these breaches, like radically larger. So if you look at the Uber breaches, for example, and I'll, I'll pick on them and I'll name them because they weren't very forthcoming about them. Uh, they apparently had credentials out there, uh, for, in a GitHub repo that, that were useful in production for a couple of years. And the, the data blast radius was quite quite large. So I think that, that, that notion of like, we can understand the risk and the blast radius, uh, might be getting a little obsoleted. And, and that's something I would just, just warn people about that as you're, as you're moving to these cloud environments, and you're moving to, uh, these modern, uh, infrastructure environments, you really, you don't have the kind of segmentation based on TCP IP networks that were there in the data center.

00:30:22

Yeah. And, uh, this maybe intersects with one of the questions that came in, which is, um, you know, when you look at these cloud environments, you know, there are because you don't have those traditional tools, they provide other tools, you know, AWS for example, has a set of capabilities around, uh, you know, securing the infrastructure there. So the question is, um, to what extent are those tools insufficient? You know, what are the gaps, um, why not just use this cloud native tools that come with your, uh, environment?

00:30:50

Well, um, there's a reason I left AWS to found fugue. Uh, so, uh, I would argue that, um, AWS builds building blocks. Um, and if you look at their own diagrams, their own documentation of how to integrate the dozen or more services they have for security, um, it is non-trivial, uh, and requires deep, deep domain expertise. Uh, there are also a lot of things that the cloud service providers themselves just don't do fully because their, their business is selling cloud capacity, right? That is, that is fundamentally what they're trying to do is, uh, is sell utilization. They want to accelerate utilization. So there are, for example, a lot of default behaviors in cloud that are, uh, from a security perspective, seem nuts, right. And, or are nuts. So for example, when you build a VPC network in AWS, if you don't specifically define what the egress rules are for that network, the default you get is egress on all ports to anywhere is, is fine.

00:32:03

We all know that that's a really bad idea from a data exfiltration perspective, but that's the default you get. So, and that shows, and I, you know, I love AWS. I worked there. It's awesome. But you have to understand that their, their motivation, of course, they want you to be secure and safe, but they also mostly primarily want to sell you compute services and data services and so on. So the other place I've seen, uh, you know, we have a lot of few, we manage just millions of resources for, uh, folks in the cloud across clouds. And that's the other aspect is most organizations at scale, they're not building single applications typically that span cloud providers, but they will have applications that are in different cloud providers at this point, uh, between the big three, at least. And there you need some way to understand your security posture across them. And none of the cloud service providers are, are motivated to, uh, to, to do that for their competitors. So,

00:33:09

Yeah. Um, Tracy, anything come to mind when it comes to sort of the behavioral analysis that, that you can do in some of these cloud, because it's done that maybe leave gaps types of

00:33:21

Absolutely. Yes. Um, and that, I mean, it continues to go back to, you know, the information war, what do you know versus what do they know? You know, um, and, uh, you know, I was reading in some of these questions, uh, the follow-up, um, important to at least know which vulnerabilities you might have in your chain stack. Yeah, absolutely. I mean, that's the unfortunate thing about the scanning for vulnerabilities and looking for CVEs and things like that, you know, that is just an it debt, um, journal wheel, right. Or we're constantly spinning on this wheel of trying to remediate vulnerabilities and update libraries and patch and, and all these things that we have to do. Um, and that you have to do those right. There is, you know, um, I often use an analogy it's like a rear view mirror in a car, right? If you rented a car and it was missing that mirror, you would probably go back and get a different car.

00:34:19

Right. It's important. It's a safety thing. We have to scan and fix vulnerabilities. Um, but then we have to go, another level of, you have to assume, you know, before those CVS are ever publicized, they exist in environments all over the world, right before you knew they were even there. So you have to go beyond just kind of this historical or medial approach to it, debt and, and fixing vulnerabilities. And you have to start looking at the behavior of your applications, getting back to some of the things around zero trust environments, again, not necessarily blocking everything, but just being able to squelch all of the normal behavior that you know, and trust and that you've seen in development and QA and staging, and, and then in prod as part of that pipeline, um, if you can establish that behavior and you can know that behavior, it makes it so much easier to identify the clues of anomalous behavior, network connections, bash running on containers, container, escape, you'll start to see those things. Um, so that's, you know, that's another approach that we have multiple layers. We need multiple tools that can do these things. And, and that's the only way you can is to be able to establish that known versus the unknown. Yeah.

00:35:32

And I think what you, what you just said answers to the second part of that question, which is, yeah. What about these unknown unknowns? You know, how do you address that? And, you know, if you're, if you're not assuming anything about the level of compromise that some component of your interaction with that sort of zero trust approach, right. Then that's, that's one approach to saying, you know, it doesn't matter what I don't know about this, this surface.

00:35:54

Exactly. We, don't always, can't always judge whether it's good or bad from the beginning. Uh, but understanding that it's anomalous, whether it's an internal actor, external, whatever the case, we have to be able to squelch down, uh, to, to have enough, you know, visibility, you can find the needles, uh, much easier in that smaller haystack. Okay.

00:36:16

Yeah. It's, it's, it's really interesting to hear, uh, other, you know, technologists talk about this in their kind of fields, because there, there are similarities and there are differences in cloudness configuration. Uh, if you, if you, if you're noticing behavior it's too late, uh, so that's one of the in, in networks that is exactly what you want to do. And Tracy, I'm totally agreeing with you there. I'm talking about if you're, if you're, if you're talking about data exfiltration out of object storage, for example, there is no traversal of packets over a customer viewable network. All there are, are get statements in a weblog really, and, and those objects stores are really good at serving content. So there are, there are classes of, by the way, I completely agree about the unknown problems. You know, the, the, the things that you haven't predicted and watching behavior as a way to, uh, to try to notice.

00:37:16

So one example of that in the sort of clouds, uh, quad resource areas, as opposed to maybe the container and network area that Tracy's an expert in, and I'm not, is changes to machine identities, uh, in, in the cloud, uh, identity and access management IAM, which is what two of the three major vendors call it is really how you build trust, relationships and networks between, uh, the components and the infrastructure. And if something is changing its uh, its identity or, or the allowances that has, that is usually something to look at right away, because, because that is effectively lateral network movement or, uh, you know, uh, privilege escalation and, and a lot of folks are still not aware that that kind of identity based network is such a fundamental part of, of, um, uh, of cloud security.

00:38:17

Yeah. Thanks. Um, we have, uh, uh, another question here, um, asking specifically about embedded systems, which I think is interesting, um, to think about, you know, I mean, it strikes me that there you have, uh, maybe more knowledge of the environment it's less sort of out, out there in the Oakland and leave your control. On the other hand, uh, the supply chains there be fights, you know, involved like complex, you know, with IO, IOT and others from embedded systems, you often find that you're running much more open source software than you realize. Uh, you would guess that a smart plug could involve, you know, so many, uh, open source packages, right? So, uh, I don't know, Brian, maybe you have some comments on, uh, you know, software bill of materials. I CA as it relates to the embedded space,

00:39:08

The embedded space is its own little microcosm, isn't it? Um, I think, uh, that's an area that is probably most invisible to their downstream consumers. I mean, when you're talking embedded, you're talking about embeddable medical devices or medical tools in hospitals, not just your cameras and, and, uh, smart plugs and you know, all these kinds of things. So, uh, some of these, these, uh, embedded devices have, uh, significant impacts and, you know, because they're typically distributed as hardware, uh, people, you know, uh, people are used to looking at, uh, physical goods, they buy and inspecting the build quality, you know, take a car out of the doors sound, how does it look? How's the panel fit, but you can't see what's in the software. Um, and the software probably matters more if, if, uh, the airbags don't deploy, cause the computer never told it to then whether the door sounds hollow or not, and yet, you know, nobody's paying attention.

00:40:06

So, um, I think in those instances, uh, because, uh, because the software is usually distributed on hardware, the bill of materials is even more important. Um, you have, it's difficult enough to inspect, you know, uh, a typical commercial package and figure out what's in it and what might be, you know, what vulnerabilities it might be dropping on your network when you deploy a new piece of software, uh, it there's zero visibility into the hardware part of it, right? And so for that reason alone, I think the, the software bill of materials and some of those initiatives are going to become, um, even more profound, the trick of course is always going to be, you know, trust, but verify probably the only thing you can do is trust in this instance. Um, so that's something I think we're going to have to grapple with, um, as an industry.

00:40:57

Um, we had another question about where do you see DAS going in the future, specifically scaling it and speeding up the run times. Um, and you know, when one thing that comes to mind areas, uh, you know, I think this trend of, of relying on cloud more for development, for testing, for security, for scanning, you know, these cloud based services, um, uh, outdated versions of services that were traditionally savant on-prem offerings, I think opens up a lot of opportunity there to do optimization on the tool side, to, you know, scale horizontally in a way that's maybe not feasible for an company it's not cost effective. And so I think that's one thing I see there, you know, I mean, you see it happening and testing that was still like testing a mobile apps and, you know, you could test across, you know, tens or hundreds of devices to really help coverage. I think based services really find that capability to scale that up. I think we'll see the same thing happening that comes to DAS. Did others sort of combine Funtime oriented security measures?

00:42:02

I mean, the application boundary is changing radically, right? And the more cloud native you get. So few for example, is mostly built out of functions as a service, uh, mostly out of Lambda on AWS and, uh, the way that, um, I tend to think of the way I think of the cloud is, uh, you know, Thomas Watson, uh, uh, there is a rumor he said in the forties that the, the global market for computers might be five. Um, in some ways he wasn't maybe wrong if he said that there there's the AWS computer, there's the Microsoft computer, the Google computer, and these are giant distributed computers. And when you think about the application boundary, when you're starting to talk about, um, uh, truly distributed architectures, which is, you know, the cloud, uh, unleashed, uh, uh, uh, uh, a billion or a million, uh, uh, distributed application architects on the world unwittingly, uh, for better or worse, that's what we're building now.

00:43:06

And when you think about that application boundary, it's not contained by operating systems anymore. It's, it's really spread across, um, this big global computer. And so what, where, where does infrastructure end and application begin? These become really interesting questions from a tooling perspective when you're thinking about how to knock down, knock these problems down to manageable chunks. Um, I think we're just starting, uh, to, to get our hands around that. And it's going to be an, an interesting ride. Um, the thing that excites me most is that because it's all software defined, like real cloud is API, right? There's no human in the loop for real cloud services. And that means we can use all the knowledge of, uh, computer science and software engineering we've developed over decades to, um, to evidence correctness, to do all kinds of good things that, that we, we just simply couldn't do, uh, when there was some, uh, some guy like me who like makes a lot of mistakes, shoving stuff in racks, you know, that was directly in that loop.

00:44:20

Um, all right. We got a couple of very specific questions that I wanted to, to get to, um, going back to our discussion of the dev environment and staging and how you, how you sort of lock that down. There's a question of specifically what kind of security automation would you put in place around a Jenkins server and build servers?

00:44:41

Yeah, I'll take that one because the obvious part, I think, is, is all the things you would think about in a normal production scenario. And I think most people would tend to treat it that way. That's obvious what's less obvious is that Jenkins servers building code, um, which in some ecosystems, uh, the installation of the dependencies is executing scripts. There's an injection point for, for a remote code execution. Um, the dependencies, uh, it's running unit tests, it's also executing that stuff, right? So I think the thing that everybody has to recognize the new attacks here are not necessarily trying to go in the front door, if you will, of Jenkins and, and leverage, uh, misconfigure in, in that API, although that certainly happens, but that's a, that's a traditional, uh, production runtime kind of problem. Right? What, what I'm seeing are more, you know, these upstream supply chain attacks, which are intended on getting the dependencies into the thing Jenkins is building, and then, you know, as it installs, as it runs a unit test, that's where the attack is happening.

00:45:46

And I think that's the, that's the entry point that most people don't think about because they're thinking, well, it's our code. We have to trust our developers not to do, you know, silly things, um, or we'll never get anything done. Yeah. But, uh, what about those 10,000 other developers that you don't know the names of, or, uh, the motivations of that represent all the open source software that you're pulling? And for the most part, it's not, it's not the people doing the open sources software. There are very few cases that I'm aware of where somebody on the project did something nefarious, it's their credentials getting stolen. Um, it's, um, you know, commits that are being put in that, that, uh, introduce intentional bugs that are very hard to detect. Um, it's, or it's just the more blatant thing. They, they put a Typosquatting component out there, um, that, you know, it's hard for the developers to figure out which one's the real one.

00:46:39

They inadvertently pull it in. And just that act right there gets code executed on their system. And we've seen just within the last month or so, uh, some of those attacks trying to flip back doors on developer machines. Now, if that isn't caught and gets committed and then runs on the Jenkins server, the same thing's going to happen there. Right? So these are the new attacks that we're talking about. So of course the old school, you've got to make sure your servers are protected in the traditional way, but these servers are running code from people. You never thought of that's the, that's the eye opening moment that I think should have people reevaluating how they're thinking about this.

00:47:14

Yeah. I was looking, uh, as I was looking in the slack, um, I noticed, uh, Brian fencer had the U S air force platform. One, that's a great example of what the us air force and the DOD are doing as far part of the platform. One approach, which is, you know, when you think of a government organization, that's huge and hard to, you know, to implement security across, uh, all those different organizations and projects and everything that's happening, you know, it's, it's kind of hard to even comprehend. So they, they taking the approach of pre hardening, pre scanning. Pre-vetting, you know, not only vendor applications, but open source tools and the operating systems and everything that's used to build containers, et cetera. I am pre-vetting those so that they have, uh, a certificate to field within the DOD that they can assure everybody, Hey, coming, come and use the tools that are out of this pot, because we've already scanned these. And we've already made sure that these are all good. You know, that's a fantastic approach. And many companies are probably doing that already. You know, having infrastructure teams kind of pre vet things, but, you know, it's a team effort and, and sometimes you have to do some of those early steps of preventing those things before you even consider bringing them into your, uh, environment.

00:48:34

Yeah. And I, I think you might, you know, as you, as you reevaluate the truth bomb that I just dropped around, what's really happening. Um, you know, you, you probably need to reevaluate how you approach the blast radius like Josh was talking about on these things and understanding how you might be able to minimize the blast radius, both on the developer machines and on the CGI infrastructure, you know, things like making sure that the part that's running unit tests, as an example, can't get to prod, you know, maybe, you know, because if you were kicking off the, the Terraform scripts, let's say from the Jenkins build automatically, and it's the same one, that's first touching all of these things. That seems like a, not a good way, you know, the separation of concern. Maybe you need to have another one. Uh, and this is back to what Josh was talking about.

00:49:21

It's, it's more sophisticated credential management at the end of the day. It may also mean, uh, something that, um, you know, it's maybe a little bit harder to pull off on the developer environment, but containers or VMs, or even just building in the cloud so you can control and quarantine those environments off. I think that's where this is going to have to go. And, you know, I think the mental model is like, Hey, if I, if, if you didn't know me at all, except from some email on the internet, and I said, Hey, I want to send you some code. Uh, I've got, I'm going to have you build it and run it. Um, and you absolutely had to do that. How would you approach making yourself safe? It's probably gonna look a lot like what security researchers do when they're investigating viruses. Right. So I think that's, that's unfortunately where this goes, but, um, you know, anything other than that leaves these doors, I think unclosed,

00:50:15

I'll just try one thing. I'll note on that, which is don't move your production data into dev or test, please

00:50:24

We'd see this.

00:50:25

I mean, it's just maybe an obvious thing to say, but it still happens. It's still bad, but don't do that one.

00:50:32

Yeah. Good, good advice. So yeah, a lot to think about your, and, um, you know, I think what Brian was talking about, some highlights to the challenge for tools of this space against firms to try and make these sorts of, uh, more rigorous approaches to security, you know, low French dinner. Uh, we got a question, uh, maybe is a good one to end on here. Uh, I'd be interested to hear what each of you see as your product strengths. So maybe we can each take a couple of minutes and, uh, talk through that. Um, maybe starting with you, Tracy.

00:51:06

Yeah, sure. Um, our product strength, uh, with new vector and, and with nexus, Sonatype our, our partnership with, with Sonatype, uh, with nexus container, um, our strength is being able to do that behavioral advantage, uh, behavioral analysis, using a network traffic as a source of truth and using container processes as a source of truth. Um, there are many sources of truth that you can use for monitoring applications and doing security, um, and being able to use, uh, actual network traffic, uh, validating that at layer seven, um, being able to see network attacks as they're coming in from that validation, you know, that's the capability that, that, uh, new vector was built to give back to developers back to operators, um, that they were losing when they started using container orchestrators like Kubernetes and OpenShift and, and platforms like that. So, uh, that's our primary strength, you know, that ability to use network traffic as our source of truth. That means we can build security policies automatically based on that behavior. Um, we're not using the, like the settings or, uh, other sources of truth to build those policies. We're using the, you know, the best source of truth, which is that network traffic and that's guarantees that we can be accurate with those policies and then enforce those policies, or at least create those zero trust, uh, bubbles. So that's, that's our, our biggest sources or our biggest advantage.

00:52:40

Josh.

00:52:42

Um, we're all about securing your, your cloud environments. So, uh, we have this really exciting partnership for us with Sonatype with nexus, where we're able to, uh, use our, some of our technology to scan things like Terraform templates and infrastructure as code in advance. Um, I think one of the unique advantages with phew is the ability to, uh, work throughout the entire software development life cycle. So the same policy that you're using through, uh, through nexus to scan your, uh, your Terraform template can be constantly used in production once that template has been used to actually build stuff. Um, those are some big advantages for us. Uh, also, uh, are I mentioned earlier, if you want to understand how to secure things, study, what hackers are doing, and we do a lot of that, uh, and a lot of the, the attack surface and approaches that hackers are using in the cloud is really, um, in obvious, uh, you know, and, and requires that research. So, and we bake that in into the product. Uh, so, um, yeah,

00:54:01

Yeah. Um, you know, our, um, I think our strength is that we've always approached this solution and architected our solution and the data, uh, that we, that we provide, um, as, uh, this was always a developer first problem. Um, even though for a long time legal and then security was largely funding this, we kinda knew from early days of consulting, that, that it was ultimately the developers that were going to make or break the success of this. And so we've, we've, uh, effectively architected a developer first, uh, control plane for bringing all of these types of data together. And of course, you know, the, the acquisition of news, um, brought that even further with the, the developer feedback loop, that's really native and conversational in, in, um, pull requests. Um, it makes, it makes it, um, you know, sort of the last mile, if you will, but, you know, as, as demonstrated with the partnerships we have with both new vector and fugue, you know, where we're going with this as the ability to take more wider disparate sets of data that different parts of the organization care about, but need developer compliance to effectively act upon that's what we're doing.

00:55:14

And, um, these are just two of the first, I mean, three, really, if you count muse, um, um, of the first kind of, uh, ways we're expanding the platform, but I see nearly infinite opportunities, um, for us to continue to add that. And, and the key is that, you know, uh, a lot of niche systems pushed onto developers are not going to work systems that were designed for the lawyers are rejected by developers, same thing with security, and, and that's, that's how we're differentiated. We're trying to normalize all this data, prioritize it, bring it to the developers where they are, so they can ultimately make the right decisions that gets everybody, um, safer and happier.

00:55:56

Yeah. And, uh, you know, music to my ears spoken. So it's always on, um, really delivering a great immigrant to developers and taking something that had been traditionally painful, painful process, you know, static analysis and not offering a lot of value, you know, and making it, uh, or more palatable in any even useful, you know, useful and transparent to developers. So that, so that issue is just get fixed, you know, as an ongoing development, right. And that's, that's been a huge focus of ours and, and providing that visibility into, you know, what's the impact of this are your bug rates going down? Are you fixing issues? You know, what did we see in terms of the impact of this rollout on your software security and quality? So, yeah, I think there's a, there's a lot of exciting stuff happening, you know, combining all these, um, all these areas of concern from, you know, Cody or writing to the containers that it's running in through that full spectrum and bringing it together in a way that, um, that works with the development process, right.

00:56:56

That lets people continue to have that agility, you know, going back to this question at the very beginning of the hour, you know, how do you, how do you maintain security while not impacting innovation? Um, and it's about process, it's about tools. And I think, um, there's some great solutions. Yeah. So thank you all for joining me. I, this was a great discussion. Uh, uh, we'll keep the conversation going in slack, you know, we're all on the, on the DevOps enterprise summit slacks, uh, feel free to reach out if you have further questions after this. Um, thanks for spending this time with us.