Las Vegas 2019
Managing Open Source Security at DevOps Scale
Duke Energy talks about their DevSecOps journey and how they're using Sonatype to continuously monitor their open source software at scale.
SP
Sal Padilla
Senior IT Architect, Duke Energy
Transcript
Agenda. We'll talk about here a little bit. We'll talk about, um, life before open source. If anybody remembers that. Uh, I do, but how could that be? Cause I'm only 30, uh, risks with open source, uh, high-level stats, reality, the journey from open source or from no open source, uh, current state where we are key takeaways and my favorite open discussion. I love open discussion because I get to learn from you as much as you learn from me. So who remembers life before open source, right? Yes. I'm not the oldest guy in the room. Fantastic. Life was simpler back then. Right. You wrote a ton of code. You tested it hopefully. And you deployed it probably to a mainframe. I remember that. So what did you think about using open source? Right? What were your thoughts around open source and using open source?
Exactly embedded hacker code, right. Legal to use maybe. Right. Who supports it? Right. Who's posts that open source is opensource good or bad. Is it safe or vulnerable in a spirit of Halloween? Is it a trick or treat? Right? I had one colleague ask when we were starting, we were thinking about using open source. Did I review every line of code in the open source component? Yeah, it's exactly my reaction. That was exactly my reaction. He didn't care for that back to the bedroom. Um, so I worked for duke energy. Uh, it is 150 plus years old, one of the largest power holding companies in the U S we have approximately 7.6 million customers. Um, we generate almost 50,000 megawatts of electricity. We actually have gas. We have about 1.6 million gas customers in four states and we operate six nuclear plants. So security is quite important to us actually.
So I'm the enterprise dev ops architect slash engineer slash evangelist. Probably a few of those in here. So I've been doing this for about over 30 years and it, um, and 10 of those about dev ops. So I've been doing dev ops for quite a while when it first started. So life before open source, right? Um, we had basically 0% of your code was from external suppliers, right? Open source components, open source libraries. All of your code is written from scratch. Right? Then we had this mind shift when we started seeing open source. Did it, it didn't make sense anymore to write code that for functions that's already been written, right? Why not just reuse this open source code, but what drove us to open source with open source? Of course, we didn't have to write as much code, right? We were more efficient, faster, and we could focus on the business logic, the logic, the code that really mattered, right. And all that silly code connecting to databases, right? Reading datas, reading files, that's all that was all taken care of us with open source, right? So where are we now?
Now life after open source. So now we went from zero to almost 85%, almost 85% of modern application code comes from open source libraries, which is quite a lot. Actually the average enterprise will download 500,000 components annually from about 3000 open source. Suppliers of that 30,000 will have known vulnerabilities, which is really shocking in the Java world, 146 billion download requests for Java components and 11 billion package downloads from Java script for JavaScript, from NPM, which is our new favorite library. On average developers have access to more than her access, more than 21,448. New open source components released every day since the beginning of 20 18, 20 1000 a day.
So your average developer will download six 60,000 packages about 6,000, some packages annually being developed developer. That's probably true last trick. As I tell it a lot, a little over 30,000 with known vulnerabilities a little over half half of the downloads have known vulnerabilities. Maybe this wasn't such a great idea. The good news is developers. Believe security is important. I know this to be true because I talked to them. They know is they, they, they are conscious of what security means to the company. The bad news is they don't have time to spend on it.
Who here works for company that has embraced agile, right? And what do they focus on? What features new cool features. Security is not cool. That's the problem, but it is important, but features that's all they ever talk about. So how do we protect the enterprise in this new paradigm? So everyone's familiar with dev ops, right? Dev dev ops sits between the developers and operations and provides a system for continuous integration and continuous delivery. Build it faster, deploy it faster. Dev sec ops is about building security into the DevOps pipeline, baking security into the pipeline. There's two aspects of DevSecOps. There's the static analysis within the pipeline, right? You have SAS, SAS, SAST, static, application security testing, scanning OSS, open source software security scanning, and to some extent code quality, right code quality scanning. Then there's the dynamic aspect of DevSecOps, the runtime aspect, where you have tools like DAS rasp.
I asked whack, I'm not going to talk about that part. I'm going to focus more on the tools within the pipeline, the static part, because we don't have time for the dynamic. So a typical build typical Jenkins build, right? Here's the typical built. We check out the code, we build it. We run unit tests and then we run three scans is how we do it today. This is current state. We run a code quality scan, an application static application security testing scan was checks for us, secure coding written by your developers and an open source software or software composition analysis scan from Sonatype nexus IQ.
So we run all those in parallel. You pass the quality gate, you go build a Docker image. You, you make it to the end. You're good to go. So with every project, build a project team and build manager. If you have a build manager can view the results of the OSS scan and act on the findings in the dashboard for the build. From within the build dashboard, they can click on the link to view report and go directly into nexus IQ and see more, uh, get more information on the scan results application teams and cybersecurity teams can view the scan results to and view details of a particular vulnerability if they so wish. So let's look at a sample vulnerability sample component found to have a vulnerability developer or cybersecurity analysts can view the graph of all the versions available of that component. Right? So right there, they can see what's current, which version is currently being used by the application and the highest policy threat. But wait, there's a clean version at 3, 4 1, the developer can choose that version and for their application and 3, 4, 1 has no policy threat. Use that version. There is a caveat to this though, by the way, sometimes they component may not have a cleaner clean version of that component or can be a transitive dependency and that they developer may not be able to change, but so it's not perfect, but at least the things you can fix, you can, you should fix.
If you can't remediate the vulnerability in your application, the application team can request a waiver. So our processes, the application team submits a waiver to either sourcing your compliance, sourcing and compliance can waive the finding and include documentation about the waiver. So we can make this issue go away. For example, sourcing may, may have bought the license for the, uh, component and we made wave it and then document the contract ID for the component, for the purchase in the waiver so that we can link the two systems together. So it wasn't just waived just for the heck of it.
So as you can see, we have two main governance groups. We have sourcing and compliance. I'm not sure what you guys call it in your companies, but those are the folks that work with our licensing and purchasing and cybersecurity. Everyone has cybersecurity, cybersecurity for security vulnerabilities and sourcing for licensed violations. This tool will scan for both. That is very important. The dashboard allows you to filter for either one. So the cyber security folks filter on their security components and sourcing filters on the licensing issues from a governance and per an enforcement perspective. This is what sourcing and compliance and cybersecurity look at this dashboard. And our goal is to keep this dashboard clean, right? In essence, this is our, this is our open source vulnerability management tool. This is how we keep track of all the open source in the company and where, where our, what our risk exposure is within the tool.
Uh, you can see what policies are being enforced at teams compliance and cyber security teams can see all of this, what policies are being forced. We can set alerts to compliance and cyber security. When an app team reaches a certain stage in their life cycle. When an application team tries to go to release stage, for example, the scan fails and the issue issues have to be addressed, right? You can't get to production. Ideally it would never get to this. I touched on this a little bit, but, uh, w when we talk about, um, open source, we almost always talk about security, but we always forget about the licensing. Almost we forget about licensing sometimes, but licensing is, has, could have potential huge financial impact to your company. So you got to really watch that too. It's not just security, but it's also the licensing.
So who owns remediation? So now that we know the vulnerabilities, who do we need to discuss about remediation? This dashboard allows us to identify which application was a source of the vulnerability. So we can have a discussion with an app, but imagine what if the CIO of your company asked you if the company was vulnerable to stretch to, or event stream from MPM, what would you say? Right. Fortunately, I was not asked that, but what would you say, how would you know, right at nexus IQ gives us a bill of materials inventory, and we can see if a, the component is in the ecosystem and B where is it? So now I can know, now I can ask that question. No, we don't have it. Or yes, we have it. And we've know the application it's in and we're going to fix it.
Our things we do is we're educating our developers on shifting security left, right? We're educating them on available tools that they can use while they're developing their applications and getting them into the habit of writing secure code and using open, safe, open source components. So they can see the same component details in their IDE that they can see in nexus IQ. So while they're developing, they're making intelligent choices, and hopefully it's not getting before the bill, before it even gets to build right. Other tools with our pipeline is code quality scanning. So the app teams can see on their dashboard for the app and quickly assess their technical debt.
The other scan we do to round out our DevSecOps within the pipeline is static application security testing, which focuses on secure coding practices. And top 10 among other things. So where are we now? Well, we just finished educating and training our sourcing and compliance team. We're training our cyber security team on these new tools and processes. We're educating. We're always educating and socializing these tools within the development community, and we're moving to active governance and enforcement. So you're warned now we're going to really start failing things and start making you, encouraging you to fix your applications.
So here's some key takeaways that I've experienced while doing this sourcing compliance and cybersecurity we'll have new roles and responsibilities. They may not have experience in with open source licensing. For example, I know when I worked with the compliance folks, they didn't really get how open-source licensing worked or how it got into the company. And I had to bring them up to speed on all of that. That was something they, they didn't understand the dynamics of application development, how components can just come in any time of the day. And it may have a license phone, Billy license violation, or they may not behave. They may be staffed to handle the new workload. So there is going to be additional workload. They might have the staff for it. So the, you have to account for that sourcing clients don't need to be engaged at the first occurrence, give applications time to remediate. So, uh, you find something don't go over there first day and beat them with a wet noodle. Cause they pulled in something bad. They see it, they're working on it. They may, hopefully they're finding a good component library to use and working it into their application or in their case, putting on a backlog and prioritizing it.
So how do you know your company's threat risk? Right? Without it, without tool like this, the, the last thing that I actually heard this morning in the keynote was convincing product owners. That security is more important than features has opportunities. They do not yet quite grasp the product owners, this, the, the importance of security versus features. And they're putting features way ahead, a priority rather than security. And I hear this all the time with the project teams, that app teams that I work with, like, yeah, we know about it, but our product owner hasn't told us to work on it. So we didn't fix it. That is something that I'm kind of curious. How do you, how would you deal with that? Like I'm getting to the open discussion part and that's when the questions, how would you, how would you deal with that convincing product owners to, to emphasize put more priority on a risk registries, risk registries Risk? I'm not sure if I'm familiar with that. What does that mean? P speak up, sorry.
The document, the risk. Put it down. It's in there. And for them to ignore it, they're explicitly saying if someone signs off someone signs off on it, when it blows up in their face, it's it's there.
Okay. That's a good idea. Risk registries, risk registries documented, right? They put their name on the line and their jobs, essentially. I like that. Any idea? Yes.
The early principles of agile was that a user story was only that
All
Framework level considered non a non customer value concern were actually baked in to make it into a story of the end customer. And you would work incrementally on those. Was it a safe cause of not
Functional requirements? Non-functional requirements? Yeah.
So when they had that same position of project management ceremonies around what was agile development practices, now we track everything basic, really hard. In the early days of agile, we never had these discussion because of the product owners focused on value. And we would just leave it framework level concerns into our velocity.
You weaved it in, but how did the product owner, how does the product owner say you should work on that
and you worked on the product owner is not involved in the conversation around
What
Operating system is used in your
Servers. So this other group has as much clout as the product owner to decide what the team works on,
Uh, basketball and power, agile dev ops team. It is, the team is empowered to make all the decisions that we on a budget or a percentage of the spread or something, how you, however you want to it. But that's what I'm accustomed to as well. You, you have 15% or whatever to spend on technical even decides what the technical is. That is that they want to work on because like, who are they going to ask what our subject matter experts?
Okay. So I Hare empower the dev sec ops cybersecurity, especially in compliance team as the same level of power as your product owners, or
They are, we can set it. That sounds more like they are central excellence or whatever, what they should do is get whatever it is in their head, go to fighting for policies in your organization. Like that's so that they don't like you can scale with people, but you can't scale the policies yet, as well as a role on your agile team. Dan north is, are like, people are already talking about merging that role into the team as a whole. And it's just a hat. Someone wears like certified scrum masters or honorable, and the team just shares that responsibility. So what's the incentive model for the developers to prioritize Eric here. Well, it's the same as if you've sent it just to write quality code and to teach developers to be good relatives. And you put your brain. I remember the days when there were entire QA organizations that I was working with and you through code of imprecise, like unknown quality over a, a bunch of tests. And then we just absorbed into the, into the teams. So that's and developers, weren't assets had she
Added to the team, had huge value to the quality of the code. And I think security people on the team at huge value, quality safety, where we see it actually gets sideways to your point. If you give the product over too much hour, they'll hold your product hostage, getting better. They'll add feature after feature after feature. Um, but a, a good product, exactly what you said, which is democratizing. The remember we've had, we've actually had three rough going because we've defined that there is a scrum master and there is a product of those two don't get along. It doesn't matter how good, but in places where we don't have that defined, you actually had some sense. Brian games recommend reading the story of riot games, dealing without product owners and scrum masters is the only two specialized rules under that. And it was counterproductive.
I don't necessarily disagree. I am not a big capital. A guy. Yes.
I think security has to be prioritized at a higher level. The solution level, the work management level, because the price of the product owner is in charge of backlog. What he's done in charge of the prep. So the management has to prioritize security, hire it, say, okay, I need security more than
Right. Like from this morning, bill gates. Right. Is that what I think I heard it right. Oh, I think they said bill gates. Uh, yeah. Security versus if you have to choose between security versus features, choose security. Ah, we gotta get there. We gotta get there because you saw it right. 50% as a known vulnerability. Yeah.
Anonymous, security, not Security is built and it's just incorporated. It's not a feature. Cause we all know what's the first thing that gets deprecated or kicked out. If you call ,
Can't be security versus features.
I actually had a different problem. Now there's a problem. So we work with health care data with Phi and we have, we process billions of transactions every year. Um, healthcare transactions are clearing out. Um, our info sentencing is designed for on-prem software. So, so we had an info set team that in order to make a release in this now flexing his muscles cloud development has had a period of freedom for awhile muscles and putting brakes on any release into the cloud that doesn't meet their review, which takes a minimum of two months, requires us to produce all of the waterfall style artifacts they take, oh, last minute, turn it into a bullet list of disconnected
Requirements. A
Hundred items long.
I don't know. I want to go back to there. I don't know what I want to go. I don't know that we've been there. I don't know that when we just went away with the pendulum swung the other way, um, legacy versus brownfield, we won't go there only got 30 more minutes. Any other questions? Yes.
Uh, DevSecOps. What do you say to people who say, if you have to say it that sec ops, you were doing your own.
If I have to say dev sec ops. Oh, because it wasn't built in because DevSecOps is a thing rather than just secure dev ops,
The joke is like, so then start calling business. That golf is
Because we like to make things up in our industry. We call things that we used to do 10, 20 years ago. Some other new because for marketing, no, no offense or Matt, anything time is up. I finished. Thank you.