In recent years, we have seen an increase in the number of catastrophic supply chain attacks in both open source software (such as with event-stream and recent dependency confusion vulnerabilities) and in the proprietary software world (with the SolarWinds and Hafnium exploits). Dealing with open source supply chain attacks can be particularly daunting due to the simple fact that rather than working with a single supplier (like SolarWinds), there can be dozens of suppliers (open source maintainers with commit privileges) for a single component. This means that your open source supply chain can include thousands of discrete suppliers when you consider that at least 70% of the code that makes up the average modern application is open source. To manage open source effectively, you need to have a strategy to address at scale a wide array of potential attack vectors and software maintenance issues. In this presentation, Tidelift CEO Donald Fischer will give application development leaders a frank assessment of the current state of software supply chain security, including an overview of common vulnerability types and an analysis of recent US government policy designed to secure the software supply chain. He’ll then share the best practices top organizations are using for open source software supply chain management and governance today, along with a set of immediately actionable recommendations organizations can implement as part of a comprehensive strategy for managing open source health and security.
CEO and Co-founder, Tidelift
Hi, I'm excited to be here today at DevOps enterprise summit. Let's get started. My name is Donald Fisher I'm co-founder and CEO at tide lift. And I'm excited to talk to you today on the topic of thinking upstream about white house, cyber security, executive order 14 0 2 8 bit of a wordy, uh, numeric title, but hopefully you'll get a lot out of this conversation. So what do you need to know about this white house executive order and what it means for your world? Well, let's start with some of the basics. So today I'm going to be specifically focused on how this executive order and a number of events that have occurred after it, and in its wake impact application development teams and the way that they build their software today. First, just to establish some of the facts. First thing is that in may of this year, 2021, the white house released an executive order with a series of detailed directives focused on improving the nation's cybersecurity.
Of course, we, all, many of us will recognize that, uh, this was in response to quite a number of an increasing cadence of high profile software and cybersecurity attacks. For example, the one that impacted a whole bunch of solar winds, customers and organizations relying on solar winds products late last year and early this year. And the third fact to point out is that this potentially has pretty large impacts on all kinds of organizations who use, who develop software and especially those that use open source software to develop applications. And there's more and more details and standards and requirements that are emerging every day. So this is what we want to talk about today and help you understand a little bit about what is the, what are the actual requirements? Um, what's the process that's unfolding and then try to come up with some, uh, recommend some good ways that you can approach, uh, dealing with these requirements and dealing with the reality that's, uh, that's that's happening.
So when the white house executive order first came out, a lot of organizations looked at it and sort of dismissed it initially saying, well, this doesn't really seem to be about us. We're not a United States federal agency. We don't primarily sell to the U S government. And that sort of misses one of the, you know, core intents of this federal initiative, which is not just to up-level the security profile of the us federal government, but also to Uplevel the rest of our digital infrastructure as well. I think it was said really well. Um, in this quote here, um, in so many areas of computer security, what the us federal government does first, the private sector follows when the federal government is requiring here will become likely to become the standard for all software moving forward, not just in the United States, but internationally. It's the Xpress intent of the federal government to try to set a higher bar here and raise the stakes, not just within the government and its direct suppliers, but also its indirect suppliers and really all of the critical digital infrastructure in the country.
So one of the, as I mentioned, one of the critical areas of focus in the cybersecurity executive order applies to software supply chain security, and that's really meaningful for application development teams in organizations that build, you know, software powered solutions. Now there are several provisions in the white house executive order that pertain here. One of those is that the executive order directed that the national Institute of standards and technology or NIST solicit input from other government agencies, academic researchers in this area and other organizations and market participants to try to crowdsource essentially what should the standards and best practices for improving software supply chain security be and set a specific timeline for that as well. Um, miss two is required to conduct this, uh, solicitation of input, uh, within 90 days of the May, 2021 executive order. And there are other requirements as well. So for example, the national telecommunications and information administration or NTIA was separately required by the executive order to publish a minimum requirement for what's called a software bill of materials for any software solutions being sold to the government.
And as you can appreciate, it's been a couple of months since this executive order first dropped. So now some of those deadlines are, um, coming upon us, we're past some of those deadlines and some of the details are emerging. So for example, on the NIST website today, you can find actually a very long and extensive document, uh, describing, uh, security measures, that critical software. And in many cases, any software that's being provided to the us federal government will need to comply with going forward and not surprisingly one of the key aspects of those security measures that matters for application development teams is that they're going to need to establish and maintain this baseline inventory of the components and versions of components that go into their application.
NTIA also delivered, uh, within its required timeframe that initial, uh, set of requirements for what's called the minimum elements of a software bill of materials. So a software bill of materials is fairly straightforward to, to understand it's basically an ingredients list for any software artifact. And for those who are using open source to develop applications coming up with this, uh, bill of materials or ingredients list for your application and, and testifying to some of the attributes that that software meets raises some pretty tricky questions, actually. I mean, when it comes to traditional commercially produced and sold software, it's very clear who the supplier of that is. It's the company it's whoever sold it to you, the vendor of it, right? And they're the ones that you would go to, to ask clarifying questions around, for example, the original authorship or the security practices that are being applied in the development of that software.
But when it comes to community created open source software, which comprises more and more of the applications that every organization is building these days, who is that supplier? I mean, is it the original open source creator that opened up the, for example, GitHub project and started contributing code there, if they've moved on, is it the current maintainers who are, you know, patching that if there are any for the particular piece of software or software component that you're asking about and what responsibilities do those creators really and maintainers really have? So breaking this down a little bit further and zeroing in on what are we actually asking suppliers, whoever that might be, uh, in the open source context to testify to, there are some specific requirements, um, in the executive order and it's, you know, the resulting standards and regulations that are falling out of that.
So for example, if you're building an application that includes open source, you're, you're going to be, um, asked to track and maintain a complete and accurate software inventory or bill of materials. As we've been talking about, you're also going to be asked to attest to the integrity and provenance of that software. So what provenance means in this context is the origin, where did it come from, who created it? And you're going to be as the application developer incorporating this into your solution, you're going to be asked to attest, to conforming with secure software development practices in the creation of all of those constituent elements. Again, most of those in so many applications are coming from, you know, independently authored third-party open source projects. How are you going to answer these questions and sign off on this requirement when providing your, uh, solution to the U S federal government or, or really any organization that chooses to adopt these requirements really, really tricky situation and let's face it.
I mean, the reality of how we have wound up building our digital infrastructure today, it looks something like this, a well appreciated and well distributed XKCD that shows all of modern digital infrastructure sitting on top of, for example, a project, some random person in Nebraska has been thankless only maintaining since 2003. And this really actually understates the problem. If you think about it, because most applications depend critically on hundreds or even thousands of discrete components like this, uh, directs dependencies like web frameworks or, um, you know, uh, parsing libraries that your application may use or, uh, additional projects that those first order dependencies rely on. So-called transitive dependencies for lower level, uh, you know, logic like a string processing or, uh, network communications. It's a really, really tricky problem, but this is the reality that we're starting with.
And here's some more facts. Uh, first party facts that we gleaned through a survey that Thai Lyft conducted recently, where we reached out to the actual upstream, open source maintainers of why they use projects. When we surveyed them, we asked them, how much are you getting paid for your work on these open source projects? And almost half of those maintainers reported that they're getting paid nothing for their open source work. Only a quarter of maintainers reported that they're making more than $1,000 a year for their open source work, which is really just pizza money. If you think about it, certainly in a, in, in a U S uh, economic context. And so it kind of doesn't add up the government is asking you to vouch for the security and provenance of the software. That's going into your applications that are provided to the us federal government agencies or their suppliers. And you're getting that software from these open source maintainers who are getting paid peanuts, or for the most part, nothing. Uh, where are you going to go to answer, to, to answer these questions like, and what incentive do those open source maintainers have to cooperate and do the typically tedious and laborious work, uh, to verify, uh, these standards are being met for the software that goes into your applications.
It's kind of a scary situation. And it leads to outcomes like this one where when we surveyed, uh, in this case, organizations that rely on open source software, we found that almost half of large organizations report that they are not very, or not at all confident that their open source components that are go into their applications are secure up-to-date and being well-maintained. I mean, if organizations are already reporting this, this is basically what the white house executive order and the NIST and NTIA downstream requirements are asking organizations to certify to federal government agencies and other organizations. This is kind of a broken situation that we're in right now.
So to think a little bit with you about how we can approach this in a more, uh, logical way, um, a more productive way. I want to stop and tell a little bit of a story. Um, I'm going to lean on a parable that's, uh, called the upstream parable and is popular in public health context. So there's a story associated with this and maybe come along with me on a, on a little thought exercise. So picture yourself, um, you're on a hike out in one of, uh, say a national park. Um, you've brought along one of your good friends. You've brought a pack to picnic lunch, um, that you're going to enjoy out in the, in, in those woods. And you find a great spot right along the side of a, a nice wide river. So you sit down and he starts to spread out your picnic, uh, get your sandwich ready, you know, pour a glass of a beverage.
And just as you're about to, um, uh, start, start dining you and your friend here, um, splashing and cries for help coming from the river. And when you look over behind you, you see that there's, uh, a child, a kid splashing in the river, seemingly struggling to stay afloat. So of course being great, um, people, both you and your friend jumping in the river, um, pull the kid out, pull them to safety. And just as you get to shore, you hear more shouting behind you. You look back and there's two more kids floating down the river are struggling. Um, so of course you and your friend, again, you, you sat down the first kid on the, on the bank, jumped back into the river, pull out the two more kids, get them to shore. Then just as you're getting the second and third kid to the shore, you hear more splashing, more, uh, more cries for help.
And at this point, uh, you turn around to, to jump back into the river and keep up the rescue mission. And meanwhile, your friend, doesn't come with you there, they start walking up the riverbank and you yell to them, Hey, where are you going? We got to save these kids and your friend yells back to you. I'm going upstream to figure out who's throwing all of these kids in the river in the first place. So how can we apply this upstream thinking approach to this domain when it comes to addressing the issues raised by the white house cybersecurity executive order that pertain to application development teams, how are we going to take an upstream approach to answering some of these questions around software bills and material, their security, uh, attributes, their provenance, where they come from, and how can we do that in a context where the creators of that software? So often didn't sign up to be part of somebody's some organizations supply chain.
So from my perspective, there's, there's three main, uh, questions that application development teams using open source need to go upstream for, to find answers. So one of those is how to even create a comprehensive and accurate software bill of materials for your application. And that includes not just the directed dependencies, like the web frameworks or UI frameworks that your application uses, but all of the downstream dependencies, the additional open source projects that get pulled into your application as part of the build by those, uh, direct dependencies, those are the so-called transitive dependencies. And then once you just have that list of all of the ingredients that are going into your application or service, how can you confidently attest as to the integrity and provenance of those open source components? And thirdly, getting to the security, uh, bottom line of some of this, how are you going to confidently attest to the fact that those open source components conform to secure software development practices, these are really challenging.
And if you think about it, we can only really comprehensively and practically address these questions by going upstream and working with the actual creators and maintainers of those, uh, components. Now in open source communities upstream has an additional, uh, meaning the word, uh, you know, in the public health parable, uh, the upstream parable, it's literally a river or a stream in open source communities. Uh, upstream is the place where the software comes from the, the community outside of your organization. Typically that's creating that, that software, the individual who's, um, you know, putting in the work, uh, creating and refining that software code. So one really, um, kind of basic, not exactly rocket science idea, but one that is extremely compelling, I think for facing this problem is what if we enlisted their help? What if we, um, reached out to those upstream open source maintainers and said, Hey, can we work together to ensure that we can make these kinds of tests, uh, uh, certifications and attestations around the software, you know, a lot of this, um, and can help us impact the actual facts around these open source projects.
Would you like to work with my organization to do that? Now that's a really challenging thing to do. If it's a many to many problem where every organization, a Mo of thousands of them needs to figure out a way to work with thousands of independent open source maintainers and teams who engage in their projects in different ways. What if we could create a sort of ordered way to, to approach that problem, where, um, we could gather a bunch of those open source maintainers of widely used components into a community, and then provide sort of a central place to go to ask for their help, certifying and attesting to the fact that the software meets certain standards. And in many cases, bringing the software up to the, um, uh, levels of, uh, uh, process and, uh, you know, uh, certainty around some of these issues and what if we paid them to do that, which is a solution that, that just makes sense on a couple of different levels.
I mean, for one thing, there's sort of a, an almost moral argument that these open source creators are creating so much value for downstream organizations, whether they be commercial or government, uh, or other kinds of organizations. So it makes sense that they should be rewarded, uh, in a lot of ways, but including financially for the work that they do. But there's also just like a basic, um, business argument around this. Um, if organizations that rely on the software need it to meet these certain standards and need to have confidence attesting to that, the work to do that is, is pretty tedious. Um, you know, it's, it's not the most creative part of, uh, working on open source software, uh, filling out metadata and ensuring that, you know, all the kinds of security related processes are in place. Um, it's not the most creative part. And so sometimes it's the part that doesn't get done since organizations need it to get done, to achieve their organizational goals.
In many cases, uh, economic goals, let's bring the maintainers into that system and invite them to participate economically in some of the, um, uh, benefits that accrue because of the work that they're doing. So this is a situation that just kind of works for everyone. It makes sense for everyone, um, downstream organizations can answer these previously unanswerable questions and open source maintainers. You know, they get a good reason to go and do some of this extra tedious work in a defined way to tee up these clean answers for organizations that rely on their software to be able to make attestations about the standards that it meets. So this is one of our goals, uh, at my company tide lift is to create a solution that will enable organizations that rely on open source to partner with those upstream open source maintainers in this way. And the way that you can think about this commercial service that we provide, that's called the tide lifts subscription is just as, uh, a better, more realistic way for organizations to get their arms around their open source software, supply chains, and proactively manage them using this upstream approach.
And in, so doing it solves a lot of problems, uh, for organizations generally, and specifically relating to the requirements of the white house cybersecurity executive order. It reduces the complexity of managing all these open source components cause you approach it with a standardized process, methodology and tools, and with those, uh, open source maintainers in the loop, and it means that your organization can competently make these required attestations to federal government agencies or other organizations, while also ensuring that you actually meet these, um, security licensing and maintenance standards. It all happens, um, in conjunction and in partnership with these open source maintainers that Tyler has gathered into our growing network of open source creators. And the bottom line impacts for your organizations are you're able to cut costs by pushing this problem up, seeing upstream, uh, and solving it at its source, as opposed to dragging kids out of the river.
One by one, metaphorically speaking, your, uh, application development teams no longer have to answer all of these questions themselves, uh, which means that they can focus on building the applications that, uh, you know, include the business logic and the business rationale for your organization. Um, that means they can move faster. You're going to get more, uh, applications shipped more quickly with less, um, complications. And you ensure that you're actually covering the basis that your risk is being managed, whether it comes to security licensing, um, you know, uh, provenance information that you're going to be asked to certify to having a software bill of materials. And just to reiterate central to this solution, are those open source maintainers and, you know, the way that it works when tide lift partners with these open source creators, sometimes they're often individuals, sometimes teams sometimes, uh, whole foundations around some of these open source projects.
The way that we, um, construct the system is that the more that our downstream subscribers, the organizations that rely on these components use those projects, the more income gets directed to the, uh, upstream, open source maintainers. And that creates really a positive feedback loop. If you will. Uh, the more value is received to downstream. The more, um, income is made available upstream. It means that the folks that are doing the important and yet tedious work upstream, have a good rationale for doing that and are appropriately rewarded, uh, for the contributions that they're making. Again, it's sort of a win for everybody. And those open source maintainers are just part of the overall solution. They're really the most essential part of it, but they're complimented by a couple other pieces. One of them is, is basically the result of the work that the open-source maintainers are doing with tide lift.
It's a set of pre-vetted catalogs of third-party open source components, um, maps to the standards that tide lift and its maintainers have assured that those components meet. We call that the catalogs of managed open source software. And then the third part is how you actually put that into action in your organization. How do you link this into your actual software development, uh, delivery, uh, pipeline, and that's enabled by a set of software tools. The tide lifts has built, uh, that our downstream organizations use and actually the upstream open-source maintainers also use these tools to basically ensure every time an application is using third-party open-source components, those components meet whatever standards the organization needs to meet. And there's a complete accounting for all of the software going in there that complete software bill of materials. If you're interested in getting a sense for how the title of solution to this problem space works, I invite you to come by the tide lift a website.
You can try a free trial of the tide lifts subscription. We've pre-populated it with some sample projects. So you can just get a sense for how the tool works and some of its capabilities, or you can upload an example project or a production project that you have. Um, and the bill of materials, the, um, a manifest file that contains information about the third-party open source that's going in there. And you can use that to get a real world example of how some of that, um, open source, uh, managed open source approach can help you solve these problems more comprehensively. Thanks so much for your attention. I hope that this has provided a useful, uh, grounding in the basics of the white house cybersecurity executive order and its downstream implications. And I hope it's also provided some ideas for novel ways and sort of emerging ways that you can approach this, taking an upstream mindset. I encourage you to check out a tide lift and the title of subscription as one example of that, that you may be able to put to use in your organization to solve these hairy problems today. Thank you.
Unlimited users from organization
Jason Cox's SRE Playlist
Service Level Objectivity: Improving Mutual Understanding Through the Language of SRE Accepted (MediaMath and Google)
Adam Shake, MediaMath Source; David Stanke, Google