Thinking Upstream About White House Cybersecurity Executive Order 14028

Hi, I'm excited to be here today at DevOps Enterprise Summit. Let's get started.

My name is Donald Fischer. I'm co-founder and CEO at Tidelift, and I'm excited to talk to you today on the topic of thinking upstream about White House cybersecurity Executive Order 14028. It is a bit of a wordy, numeric title, but hopefully you will get a lot out of this conversation.

So what do you need to know about this White House executive order, and what does it mean for your world? Let's start with some of the basics.

Today I am going to be specifically focused on how this executive order, and a number of events that have occurred after it and in its wake, impact application development teams and the way that they build their software today.

First, just to establish some of the facts. In May of this year, 2021, the White House released an executive order with a series of detailed directives focused on improving the nation's cybersecurity. Many of us will recognize that this was in response to an increasing cadence of high-profile software and cybersecurity attacks, for example the one that impacted a whole bunch of SolarWinds customers and organizations relying on SolarWinds products late last year and early this year.

The third fact to point out is that this potentially has pretty large impacts on all kinds of organizations who develop software, especially those that use open source software to develop applications. There are more and more details, standards, and requirements emerging every day.

This is what we want to talk about today: help you understand what the actual requirements are, what the process is that is unfolding, and then recommend some good ways that you can approach dealing with these requirements and the reality that is happening.

When the White House executive order first came out, a lot of organizations looked at it and sort of dismissed it initially, saying, "Well, this doesn't really seem to be about us. We're not a United States federal agency. We don't primarily sell to the U.S. government."

That misses one of the core intents of this federal initiative, which is not just to up-level the security profile of the U.S. federal government, but also to up-level the rest of our digital infrastructure as well.

I think it was said really well in this quote here: "In so many areas of computer security, what the U.S. federal government does first, the private sector follows. What the federal government is requiring here will likely become the standard for all software moving forward, not just in the United States, but internationally."

It is the express intent of the federal government to try to set a higher bar here and raise the stakes, not just within the government and its direct suppliers, but also its indirect suppliers and really all of the critical digital infrastructure in the country.

As I mentioned, one of the critical areas of focus in the cybersecurity executive order applies to software supply chain security, and that is really meaningful for application development teams in organizations that build software-powered solutions.

There are several provisions in the White House executive order that pertain here. One is that the executive order directed the National Institute of Standards and Technology, or NIST, to solicit input from other government agencies, academic researchers in this area, and other organizations and market participants to crowdsource what the standards and best practices for improving software supply chain security should be. It set a specific timeline for that as well. NIST was required to conduct this solicitation of input within 90 days of the May 2021 executive order.

There are other requirements as well. For example, the National Telecommunications and Information Administration, or NTIA, was separately required by the executive order to publish minimum requirements for what is called a software bill of materials for any software solutions being sold to the government.

As you can appreciate, it has been a couple of months since this executive order first dropped, so now some of those deadlines are coming upon us. We are past some of those deadlines, and some of the details are emerging.

For example, on the NIST website today, you can find a very long and extensive document describing security measures that critical software, and in many cases any software that is being provided to the U.S. federal government, will need to comply with going forward. Not surprisingly, one of the key aspects of those security measures that matters for application development teams is that they are going to need to establish and maintain this baseline inventory of the components and versions of components that go into their application.

NTIA also delivered, within its required timeframe, that initial set of requirements for what is called the minimum elements of a software bill of materials.

A software bill of materials is fairly straightforward to understand. It is basically an ingredients list for any software artifact. For those who are using open source to develop applications, coming up with this bill of materials, or ingredients list, for your application and testifying to some of the attributes that software meets raises some pretty tricky questions.

When it comes to traditional commercially produced and sold software, it is very clear who the supplier is. It is the company. It is whoever sold it to you, the vendor. They are the ones you would go to to ask clarifying questions around, for example, the original authorship or the security practices that are being applied in the development of that software.

But when it comes to community-created open source software, which comprises more and more of the applications that every organization is building these days, who is that supplier? Is it the original open source creator that opened up, for example, the GitHub project and started contributing code there? If they have moved on, is it the current maintainers who are patching that, if there are any, for the particular piece of software or software component that you are asking about? And what responsibilities do those creators and maintainers really have?

Breaking this down a little bit further and zeroing in on what we are actually asking suppliers, whoever that might be in the open source context, to testify to, there are some specific requirements in the executive order and the resulting standards and regulations that are falling out of that.

For example, if you are building an application that includes open source, you are going to be asked to track and maintain a complete and accurate software inventory or bill of materials. You are also going to be asked to attest to the integrity and provenance of that software. Provenance in this context means the origin: where did it come from, who created it? And as the application developer incorporating this into your solution, you are going to be asked to attest to conforming with secure software development practices in the creation of all of those constituent elements. Again, most of those in so many applications are coming from independently authored third-party open source projects.

How are you going to answer these questions and sign off on this requirement when providing your solution to the U.S. federal government, or really any organization that chooses to adopt these requirements? It is a really, really tricky situation.

Let's face it: the reality of how we have wound up building our digital infrastructure today looks something like this well-appreciated and well-distributed XKCD webcomic that shows all of modern digital infrastructure sitting on top of, for example, a project some random person in Nebraska has been thanklessly maintaining since 2003.

This really understates the problem, if you think about it, because most applications depend critically on hundreds or even thousands of discrete components like this: direct dependencies like web frameworks or parsing libraries that your application may use, or additional projects that those first-order dependencies rely on, so-called transitive dependencies for lower-level logic like string processing or network communications. It is a really, really tricky problem, but this is the reality that we are starting with.

Here are some more facts, first-party facts that we gleaned through a survey that Tidelift conducted recently where we reached out to the actual upstream open source maintainers of widely used projects.

When we surveyed them, we asked them, "How much are you getting paid for your work on these open source projects?" Almost half of those maintainers reported that they are getting paid nothing for their open source work. Only a quarter of maintainers reported that they are making more than $1,000 a year for their open source work, which is really just pizza money if you think about it, certainly in a U.S. economic context.

So it kind of does not add up. The government is asking you to vouch for the security and provenance of the software that is going into your applications that are provided to U.S. federal government agencies or their suppliers, and you are getting that software from these open source maintainers who are getting paid peanuts, or for the most part nothing.

Where are you going to go to answer these questions? What incentive do those open source maintainers have to cooperate and do the typically tedious and laborious work to verify that these standards are being met for the software that goes into your applications?

It is kind of a scary situation, and it leads to outcomes like this one, where when we surveyed organizations that rely on open source software, we found that almost half of large organizations report that they are not very or not at all confident that their open source components that go into their applications are secure, up to date, and being well maintained.

If organizations are already reporting this, this is basically what the White House executive order and the NIST and NTIA downstream requirements are asking organizations to certify to federal government agencies and other organizations. This is kind of a broken situation that we are in right now.

To think a little bit with you about how we can approach this in a more logical way, a more productive way, I want to stop and tell a little bit of a story. I am going to lean on a parable that is called the upstream parable and is popular in public health contexts.

There is a story associated with this. Come along with me on a little thought exercise. Picture yourself: you are on a hike in, say, a national park. You have brought along one of your good friends. You have packed a picnic lunch that you are going to enjoy out in those woods, and you find a great spot right along the side of a nice wide river. You sit down, start to spread out your picnic, get your sandwich ready, pour a glass of a beverage, and just as you are about to start dining, you and your friend hear splashing and cries for help coming from the river.

When you look over behind you, you see that there is a child, a kid, splashing in the river, seemingly struggling to stay afloat. Of course, being great people, both you and your friend jump in the river, pull the kid out, pull them to safety, and just as you get to shore, you hear more shouting behind you. You look back and there are two more kids floating down the river, struggling.

So of course, you and your friend again set down the first kid on the bank, jump back into the river, pull out the two more kids, and get them to shore. Then just as you are getting the second and third kid to the shore, you hear more splashing, more cries for help.

At this point, you turn around to jump back into the river and keep up the rescue mission. Meanwhile, your friend does not come with you. They start walking up the riverbank, and you yell to them, "Hey, where are you going? We have got to save these kids." Your friend yells back to you, "I'm going upstream to figure out who's throwing all of these kids in the river in the first place."

So how can we apply this upstream thinking approach to this domain when it comes to addressing the issues raised by the White House cybersecurity executive order that pertain to application development teams?

How are we going to take an upstream approach to answering some of these questions around software bills of materials, their security attributes, their provenance, where they come from? How can we do that in a context where the creators of that software so often did not sign up to be part of some organization's supply chain?

From my perspective, there are three main questions that application development teams using open source need to go upstream for to find answers.

One is how to even create a comprehensive and accurate software bill of materials for your application. That includes not just the direct dependencies like the web frameworks or UI frameworks that your application uses, but all of the downstream dependencies, the additional open source projects that get pulled into your application as part of the build by those direct dependencies. Those are the so-called transitive dependencies.

Then, once you have that list of all of the ingredients that are going into your application or service, how can you confidently attest as to the integrity and provenance of those open source components?

Thirdly, getting to the security bottom line of some of this, how are you going to confidently attest to the fact that those open source components conform to secure software development practices?

These are really challenging, and if you think about it, we can only really comprehensively and practically address these questions by going upstream and working with the actual creators and maintainers of those components.

In open source communities, upstream has an additional meaning. In the public health upstream parable, it is literally a river or a stream. In open source communities, upstream is the place where the software comes from: the community outside your organization, typically, that is creating that software, the individual who is putting in the work, creating and refining that software code.

One really basic idea, not exactly rocket science but extremely compelling for facing this problem, is: what if we enlisted their help? What if we reached out to those upstream open source maintainers and said, "Hey, can we work together to ensure that we can make these kinds of certifications and attestations around the software? You know a lot of this and can help us impact the actual facts around these open source projects. Would you like to work with my organization to do that?"

That is a really challenging thing to do if it is a many-to-many problem where every organization, thousands of them, needs to figure out a way to work with thousands of independent open source maintainers and teams who engage in their projects in different ways.

What if we could create a sort of ordered way to approach that problem, where we could gather a bunch of those open source maintainers of widely used components into a community and then provide a central place to go to ask for their help certifying and attesting to the fact that this software meets certain standards, and in many cases bringing the software up to the required levels of process and certainty around some of these issues?

And what if we paid them to do that? That solution just makes sense on a couple of different levels. For one thing, there is almost a moral argument that these open source creators are creating so much value for downstream organizations, whether they are commercial, government, or other kinds of organizations. It makes sense that they should be rewarded, including financially, for the work that they do.

There is also a basic business argument. If organizations that rely on the software need it to meet certain standards and need to have confidence attesting to that, the work to do that is pretty tedious. It is not the most creative part of working on open source software: filling out metadata and ensuring that all kinds of security-related processes are in place. It is not the most creative part, and so sometimes it is the part that does not get done.

Since organizations need it to get done to achieve their organizational goals, in many cases economic goals, let's bring the maintainers into that system and invite them to participate economically in some of the benefits that accrue because of the work that they are doing.

This is a situation that just kind of works for everyone. Downstream organizations can answer these previously unanswerable questions, and open source maintainers get a good reason to do some of this extra tedious work in a defined way, teeing up clean answers for organizations that rely on their software to be able to make attestations about the standards that it meets.

One of our goals at my company, Tidelift, is to create a solution that will enable organizations that rely on open source to partner with those upstream open source maintainers in this way. The way you can think about this commercial service that we provide, called the Tidelift Subscription, is as a better, more realistic way for organizations to get their arms around their open source software supply chains and proactively manage them using this upstream approach.

In so doing, it solves a lot of problems for organizations generally, and specifically relating to the requirements of the White House cybersecurity executive order. It reduces the complexity of managing all these open source components because you approach it with a standardized process, methodology, and tools, and with those open source maintainers in the loop.

It means that your organization can confidently make these required attestations to federal government agencies or other organizations, while also ensuring that you actually do meet these security, licensing, and maintenance standards. It all happens in conjunction and in partnership with these open source maintainers that Tidelift has gathered into our growing network of open source creators.

The bottom-line impacts for your organizations are that you are able to cut costs by pushing this problem upstream and solving it at its source, as opposed to dragging kids out of the river one by one, metaphorically speaking.

Your application development teams no longer have to answer all of these questions themselves, which means they can focus on building the applications that include the business logic and business rationale for your organization. That means they can move faster. You are going to get more applications shipped more quickly with fewer complications, and you ensure that you are actually covering the bases, that your risk is being managed, whether it comes to security, licensing, provenance information that you are going to be asked to certify to, or having a software bill of materials.

Just to reiterate, central to the solution are those open source maintainers. When Tidelift partners with these open source creators, sometimes they are individuals, sometimes teams, sometimes whole foundations around some of these open source projects. The way we construct the system is that the more our downstream subscribers, the organizations that rely on these components, use those projects, the more income gets directed to the upstream open source maintainers.

That creates a positive feedback loop. The more value is received downstream, the more income is made available upstream. It means that the folks doing the important and yet tedious work upstream have a good rationale for doing that and are appropriately rewarded for the contributions that they are making. Again, it is a win for everybody.

Those open source maintainers are part of the overall solution. They are really the most essential part of it, but they are complemented by a couple other pieces. One is the result of the work that the open source maintainers are doing with Tidelift: a set of pre-vetted catalogs of third-party open source components mapped to the standards that Tidelift and its maintainers have assured those components meet. We call that catalogs of managed open source software.

The third part is how you actually put that into action in your organization. How do you link this into your actual software development delivery pipeline? That is enabled by a set of software tools that Tidelift has built that our downstream organizations use, and that upstream open source maintainers also use, to ensure every time an application is built using third-party open source components, those components meet whatever standards the organization needs to meet, and there is a complete accounting for all of the software going in there: that complete software bill of materials.

If you are interested in getting a sense for how the Tidelift solution to this problem space works, I invite you to come by the Tidelift website. You can try a free trial of the Tidelift Subscription. We have pre-populated it with some sample projects so you can get a sense for how the tool works and some of its capabilities.

Or you can upload an example project or a production project that you have, and the bill of materials, a manifest file that contains information about the third-party open source that is going in there. You can use that to get a real-world example of how some of that managed open source approach can help you solve these problems more comprehensively.

Thanks so much for your attention. I hope this has provided a useful grounding in the basics of the White House cybersecurity executive order and its downstream implications. I hope it has also provided some ideas for novel and emerging ways that you can approach this, taking an upstream mindset. I encourage you to check out Tidelift and the Tidelift Subscription as one example that you may be able to put to use in your organization to solve these hairy problems today. Thank you.