Securing Upstream: Addressing the Systemic Issues in the Software Supply Chain that Led to Log4Shell

Good morning, everyone. Welcome back. I know I'm competing against snacks and networking and lots of folks who haven't seen each other for three years, so I have no problem that the room isn't as full as it was 20 minutes ago. But you all are going to get a special treat: I am going to go very fast, and I'm going to be very nerdy. Fast and nerdy with this. I've got 35 slides.

I'm going to tell you what we're doing at the Linux Foundation in a project I lead called the Open Source Security Foundation. But really this is a story about open source software and security, with a couple of interesting asides that hopefully you'll find useful.

I think I'm probably preaching to the choir here when I say open source software is everywhere. Ninety-eight percent, by recent analysis, of applications out there directly consume open source software dependencies. This is true of the broader ecosystem as well as specifically in the Android ecosystem. This is what Synopsys found. More interesting to me are the numbers that 70% to 90%, depending upon who you ask and how they ask, of the average software stack -- software sitting inside your phone, your car, the average software container deployed to the cloud -- is pre-existing open source code. Of course, the 10% to 30% is the differentiator, but open source software is like the dark matter of the software universe. It spans everything from the operating system, obviously, but increasingly down into firmware and even chipsets, all the way up to end-user applications and mobile apps.

More interesting as well has been the explosion of dependencies. Certain very popular ecosystems, like the npm ecosystem, have really driven toward a model of lots and lots of individual modules rather than libraries, driving the average number of dependencies that an application pulls in from 84 about seven years ago to at least, in 2020, 528, as measured by Sonatype's State of the Software Supply Chain report, which a new version just came out this morning. I haven't had time to update this slide, but we should check that out.

We've also started to see that not only are there software vulnerabilities -- and we'll talk about a big one, Log4j, in just a little bit, Log4Shell -- but we're also seeing new kinds of attacks on the software supply chain. Perhaps the best example was a few years ago with SolarWinds, which was really a subverted build environment. These compound the vulnerabilities in underlying source code by adding new ways for that entire software supply chain to be compromised: typosquatting, developers pulling in an ordinary type of library and instead pulling in a clone of that that does the same thing but actually adds a cryptocurrency miner behind the scenes or some other back door. A lot of these were previewed and detailed in a white paper that came out about two years ago called "The Backstabber's Knife Collection" that really helped crystallize a lot of these new kinds of attacks. We're finding new categories of these attacks all the time, and they are being exploited out there.

This is really an attribute and a logical conclusion of the fact that we developed a lot of processes in the open source space, and frankly in the software development space, at a time of high trust. At a time when you tended to pull software from vendors who had either a commercial relationship with you, or you were pulling software off open source repositories from well-known distribution points like Apache or the Python Software Foundation or others, where some of what those organizations did was invest in hardening the infrastructure and hardening the projects in the way things were distributed. But even then, those projects tended to be happy with putting binaries up on websites, and as long as it was protected with HTTPS, that was considered good enough.

Even then, when somebody showed up on a developer mailing list and said, "I've written a new module that does X, and here's the URL for it, and it would be really helpful as a pull request," there was a default bias toward trusting anonymous participants rather than really scrutinizing. Nobody wanted to look a gift horse in the mouth, I guess. But these days we can't afford that default level of trust. We can't afford that informality when it comes to how we assemble dependencies and how software moves from what's in the developer's head and what goes into their IDE out to the finished products that we consume. A lot of it has to do with the complexity that comes from pulling in not just maybe a dozen libraries as you are building an app, like maybe happened in the early days of Apache when we were working on it, to what is today hundreds and hundreds of dependencies, including implicit ones a couple of layers down. Those kinds of social processes simply don't scale anymore.

Then a really big thing happened about a year ago. I imagine many of you, most of you in this room, had your winter holidays ruined by a particularly nasty incident called Log4Shell, which was about as close to an industry-wide four-alarm fire as we have when it comes to dealing with a vulnerability in a piece of software. I imagine most of you might not even have realized you were running it. Certainly most of you did not know where you were running it or what other software depended upon it.

It was really kind of a perfect storm. I won't read this whole timeline, but we're talking about about a month and a half from the time a researcher, working for Alibaba, first discovered the first vulnerability in it to the flubbed fix to that, that mentioned an undisclosed CVE number, that sent everybody's red flags up and caused this mad scramble and made it unfortunately very difficult to do a coordinated vulnerability disclosure release. The fix they ended up working on ended up having its own bugs. The scramble led people to discover other related bugs in Log4j. All told in those six weeks, about four different CVEs came out, as well as a series of updates to Log4j. Each one was a critical patch, each one required these downstream effects of people who bundled this stuff, of people who got software from vendors, this mad dash.

One funny example was people sending faxes to the Apache Software Foundation demanding that they certify that their own use of Log4j wasn't affected by the bug. People were unclear on the concept. But at the end, among other things within those six weeks, on January 4, 2022, the FTC issued a bulletin that told people: if we find that you have a data breach, we're going to fine you as we always do, and if we find that part of the reason you had a data breach was that you had an unpatched version of Log4j, we will add fines to that fine. That window of time, just imagine how short that was, goes to show how critical an issue this became.

It raised important questions for people. Is open source software's reputation for generally having decent security, enough security to be able to run sight unseen inside critical infrastructure, well deserved? Does the Log4Shell incident demonstrate deep and pervasive technical issues with how we consume and develop open source code? Do these issues extend to the sustainability model for open source code? If there were things that weren't done, why weren't they done? Who was supposed to do those things? Who had the obligation to do those things? And if this is a massive collection of infrastructure built by volunteers, and we can talk about the word volunteer, can we really depend upon it to run water systems and electrical grids and naval warfighting ships and things like that?

It wasn't just devs and CTOs and CISOs asking these questions. It was increasingly compliance and risk officers. It was the cybersecurity insurance industry. It was and is the White House's National Security Council, the FTC, the UK's cybersecurity agency, and many other governments, a lot of whom came to us to have a conversation as well.

Part of the answer to those questions came in the form of a report several months later by the Cyber Safety Review Board on Log4j. This was kind of like when an airline crashes and the National Transportation Safety Board forms a panel to figure out the cause of that crash, aside from sudden loss of altitude: what was the root cause? This was very much a panel brought together under CISA to look at the root causes leading to Log4j. They put out a per-page high-impact, high-insight report. It was only about 50 pages long, looked at all these different contributing factors, found it to be somewhat of a perfect storm, and is available for free download.

What they found was that because Log4j was so widely embedded and people took it for granted, frankly, it was difficult to identify using common scanning approaches. Often it would be embedded inside of JAR files, so simply looking through a file directory listing, you wouldn't know that you were running Log4j because it was in a JAR file somewhere. The Log4j framework is incorporated into thousands of software components globally, and the bug affected everything from Apple iTunes to people badging into security systems to get physical access to warehouses. It was such an expansive bug.

Consistent with industry convention, the Log4j and Apache Software Foundation teams do not manage or really know who uses the software they produce. There's no telemetry. There's no contractual relationship. There's no backlink to the users. They don't even know how widely it's used except by studies that get done by folks like Sonatype or other groups, and even then, if you measure downloads, that's just one side of the picture. Finally, the Log4j vulnerability was easy for attackers to exploit and affords a high degree of system control. That was an unfortunate attribute of the role Log4j played and some features people added to it that went unmaintained.

The report further said what Log4j really needed, what that community really needed, was better training in secure coding practices. They needed security-oriented design reviews. There wasn't really anybody who sat down comprehensively and said, did you realize that you put this chainsaw called JNDI LDAP resolution of entities from the log file into the middle of a dinner cutlery set, and some people might get hurt when they reach for that and try to use it in incorrect ways? There was nobody really thinking about threat models on the project, and there was no formal security audit that had been done on that code. We can forgive them for that. Security audits are the kinds of things that certain well-resourced projects can afford to do. It generally costs, for something at the level of complexity of Log4j, on the order of $100,000 to do a decent job looking not just for one-off-by-one security errors that might lead to memory corruption issues, but also to really look at the architecture. It's expensive, and a group of volunteer developers, even though a lot of them work for interesting corporations, can't necessarily mobilize those funds to do a spot audit at any certain point in time.

The CSRB also found that the only way to reduce the likelihood of risk to the ecosystem caused by vulnerabilities in Log4j was to ensure that the code was developed according to industry best practices. What are those industry best practices? Anybody who's taken software development courses learns how to build things, but you don't necessarily think about how to attack them. Very rarely in computer science courseware, whether at a university, in informal hackathon training, or as most people learn programming by looking things up on the internet, do you get that systematic training on how to think like a hacker, how to think like an attacker on your own code, how to build defensively. You're really trained to get to the fastest answer to any problem. The volunteer-based nature of the open source community needs sustained financial support and expertise to make adopting these practices better.

Let me add a personal take as well: the Log4j developers weren't amateurs. They weren't untrained. They were working for some of the largest companies in the world. But this was one part of what they were doing in their larger roles. No one was assigned to work on Log4j. No one had that as part of their OKRs. It was a means to an end rather than a product they were responsible for shipping. They accepted a feature submission for a major new feature that wasn't well maintained by the people who brought it in, so there was no check on that, no validation that this was something that needed to be maintained as the architecture evolved or owned.

The users of Log4j took it for granted, or, to put it more benignly, they were not well informed about the risk of this module in the midst of thousands of other open source dependencies that they just pulled off the net, and consequently they weren't really well informed about the need or the potential to invest in that security. In some ways, this is the fight that the OpenSSF was formed to try to address. I have a bug here: it was established by us in 2020, not 2002. The OpenSSF is a global initiative focused on finding the investment, the resources, and the expertise to improve the security of open source software and the software supply chain.

Our premise is that open source software's reputation for high security is not undeserved, but it's very uneven. Some projects are fairly well resourced, have the luxury of lots of eyeballs per line of code, have the ability to hire security audits, were built from the beginning with security in mind, and frankly have simply been around long enough to have been attacked by a lot of people and come out of it with a better result. The Linux kernel is a 30-year-old project at this point. Other projects don't have those resources. How do we systematically address some of the root causes that lead to insecurity in open source projects? Through a combination of open source software tools, specifications work, and open education resources intended to uplift the entirety of the ecosystem.

This isn't just wishful thinking. This has involvement from the major cloud vendors, the major platform providers, financial services firms, and lots of end-user organizations. They are all involved and have people working on these different initiatives. It would take far too long to dive into all the things going on. It's a little bit of a circus, with reference to what was spoken here earlier today about the need for bottom-up architecture and community participation. We have principles that unify us, well known in the open source community, around working transparently, everything under an open source license, and an open door to participate in all these different pieces of work. These are organized by working group with a lot of different things underneath.

I wanted to give some specific things that your organizations might want to think about consuming and using, and in the course of doing that, potentially coming back and helping us work on. We recently published two concise guides, one for evaluating open source software and another for developing it. For evaluating open source software, you want to understand attributes like: where did it come from, and what was the rigor with which it was developed? Is this something that has had one or two people working on it, or is there really a team behind it? Is there a security list where you can report vulnerabilities, which is an indicator that they take security seriously, or is this something still pretty nascent that hasn't gotten that far? Are there instructions on how to report vulnerabilities? This concise guide walks you through those attributes and gives pointers to follow-up information. It's intended to help provide a human layer to the decision you might have to make when a developer in your organization says, "I want to use some new code."

Turning that back around, when your developers are submitting patches upstream or pull requests, or in some cases even launching new open source projects, there's a set of practices recommended for open source projects to follow, from using multi-factor authentication tokens with GitHub and other important resources, to a combination of tools that detect vulnerabilities throughout the CI/CD pipeline, to implementing automated tests. We've been doing a lot of work with fuzzing tools to make that more practical to use and to train developers so they don't just generate noisy output, but output they can turn into looking for bugs at a lower level.

These two guides correlate very strongly with some courseware we've developed called the Secure Software Development Fundamentals course. It's only about 15 to 18 hours long. It's not an onerous amount of work. You can take it on the Linux Foundation's website, but we also syndicate it using SCORM, if anyone is familiar with that technology, so you can host it inside your corporate training portals. We offer it free. Not only can you take the course, you can take tests and get a certificate, a badge you can put on LinkedIn or other sites that attest to the fact that you've taken the course and meaningfully understood what was being shared. This is the kind of content every software developer should learn. Had Log4j developers taken this course, they would have seen the admonition against trusting user-submitted input, even in the form of a log file, and certainly not parsing it for format strings, which is one of the things they did.

We've also developed the Best Practices Badge, a checklist that project maintainers can go through, some of which is automated, that allows them to attest to the fact that they have these attributes in projects that the concise guides look for. It makes it easier for you when you're thinking about using a piece of software: does it pass the Best Practices Badge?

There's another automated tool called OpenSSF Scorecards, which scans repositories. In fact, we've run this now on a million different GitHub repos, and it automatically looks for certain heuristics. These are not so much looking for vulnerabilities in code as things like, are you locking your dependencies, so a rogue dependency upgrade by a hacker that added a cryptocurrency miner behind the scenes is less likely to affect your use of that module? Other checks can be automated, done at scale, and come up with essentially a credit score or risk score for using software. Earlier today, Sonatype announced the release of their new State of the Software Supply Chain report, and Stephen will be talking about that tomorrow, so I don't want to steal any of his thunder. But one thing they look at is the application of those Scorecard scores to risk factors they're able to measure, and they see a high correlation between projects that score well on Scorecards and those with better actual security as measured by number of vulnerabilities and widespread use. It's cool to have that validation. There's even an effort by the Cloud Native Computing Foundation, another Linux Foundation project and home for Kubernetes and other cloud-native technologies, to use this as the basis for uplifting every project under their umbrella. This kind of systematic, ecosystem-wide effort is something we hope to see repeated in other ecosystems.

Another product that's come out of the OpenSSF is Sigstore. Sigstore is a tool for signing software artifacts through the supply chain so that your trust that a module you've pulled from a repo is actually the module as signed by the original developers is enhanced. It defeats a whole lot of supply chain attacks that have been out there. It's about to hit a 1.0 release. The cloud-native community is already using it to sign Kubernetes releases and quite a bit of the rest of cloud infrastructure.

We've also got a specification and tools for something called SLSA, which is a way to implement a DevSecOps pipeline that allows you to have levels of attestation as objects move through that pipeline on their way to production. It's very much a way to implement the NIST Cybersecurity Framework or other similar risk frameworks that are essential to being able to look at the pipeline as a whole and have high confidence that what's at the end of that pipeline is as intended by everyone who participated. We've also published a guide for coordinated vulnerability disclosure for open source software that would have helped prevent the mad scramble that happened in the instance of Log4j. Finally, we're pulling together objective data about the security attributes of existing open source projects into a common risk dashboard. You can today go to metrics.openssf.org to get a big picture of what that looks like, but we'll be collecting not just the dashboard but the raw data behind it so people can do analytics against their own use of open source code, all with the idea of helping everybody be better informed about what open source they're using and where the risk is, and using that hopefully to drive investment back into open source code.

I'm at the end of my time, so I want to end with one cool thing we did in response to the Log4j hack. When the White House called us, when the National Security Council called us and asked, "What are you doing about this?" we said, "We've got a circus here. Isn't that cool?" They said, "That's nice, but how are you actually going to close these issues?" We in the community came up with basically a prototype plan for getting all of these to the point where they can have a double-digit percent impact on each of the problems they articulated. Across ten different sectors, across ten different themes, we pulled together a $150 million plan to put into practice over the next two years that, for everything from SBOMs to code auditing to other things, we think would dramatically reduce the impact of bugs and dramatically reduce the likelihood of the next Log4j.

We've raised a little bit of that money so far, about $30 million from our own members, but we have conversations going on now with some rather large organizations for whom this might be pocket change. Because $150 million sounds like a lot, but $700 million is a bigger number. That's the number that the FTC fined Equifax for the breach in 2017. If we can just prevent one of those, obviously that's a win somewhere in the ecosystem. But I think we could prevent hundreds of these kinds of things, if not thousands or more. That's the ounce of prevention is a pound of cure. I'm way out of time. I apologize. My slides are up. Thank you very much.