Europe 2021
Are We Forever Doomed To Software Supply Chain Security?
The adoption of open-source software continues to grow and creates significant security concerns for everything from software supply chain attacks in language ecosystem registries to cloud-native application security concerns. In this session, we will explore how developers are targeted as a vehicle for malware distribution, how immensely we depend on open-source maintainers to release timely security fixes, and how the race to the cloud creates new security concerns for developers to cope with, as computing resources turn into infrastructure as code.
This session is presented by Snyk.
LT
Liran Tal
Director of Developer Advocacy, Snyk
Transcript
Hey everyone. And thank you for joining my doc. Are we forever doomed by software supply chain risks. If you're joining this doc, then it means you care about software and you care about software security. Most of all, you are curious, like I am. How does open source software supply chain security impacts all of us, you, me and everyone else. Let's take a moment to reflect on this picture. When you look at this photograph, what comes to your mind? Is it a futuristic outlook of the world? Perhaps the uprising of our robot overloads for me, I ask myself, how much does the robot learn about my child? And where is this information stored? Is it stored safely? I think about where does the robot get eight software updates from? And can this upstream source be compromised? What can happen when it gets compromised? Not just if what's the probability that someone hacks in and you can watch the entire video feed and interact with my child.
So these are some of the things that keep me up at night today. I would like to share with you real world stories of how developers play a fundamental role in a recent and growing security incidents. Why you should care about software supply chain security and leave you off to think about where do you put your trust in case you had a doubt we're seeing more and more open source software being developed. Year after year, open source software repositories are growing with more software footprint, the applications we build you and me, everyone else we are ever growing. Either dependency of open source software. More of a software engineers are accustomed to this habit to the ways of open source, software, communication and contribution. And more of us are becoming maintainers of open source software. The growth of open source software doesn't come without any risks.
We are continuously witnessing the growth of security vulnerabilities in open source, softwares, ACA systems like NPM and Java and others. These are everything from CDE based official reports of security vulnerabilities that need to be addressed by those of us who are using an impacted those package versions in our applications and others like incidents of malicious packages, heating the software supply and targeting us as developers who rely on open source software packages. This is my stories of open source. So let's rewind back in time and to get an early glimpse of how one developer perceived the risks of open source software in 1984, during award-winning Ken Thompson wrote a short essay titled reflections on trusting trust in which he describes how we added the back door to the Unix slogging program. And then he continued and added the back door to a C compiler. And then he further continued this chain of attack by backdooring the compiler that compiles the compiler.
Now in his revelation of how software can be taught to learn specific traits and pass them on to their stones. Other programs that they create, he explains how software can remain without a trace of a Trojan horse because of the dangers of trusting code that we did not entirely create ourselves. Anything from the application source code, all the way up to the compiler, the assembler, the CPU unit, the hardware, everything else, if you cannot trust it, what do you do at this point, as we were learned by Thompson's Trojan horse story dating back from 1984, developers have been targeted as a vehicle to distribute malware and back doors for a very long time. Now let's explore some of those more recent events in 2018. The JavaScript ecosystem witnessed its first high impact spearheaded surgical attack, targeting maintainers and developers working in open source and themselves being used as an attack vehicle to distribute malicious JavaScript code design, to decrypt itself and running a specific environment, targeting developers of Bitcoin wallet applications.
This was the well-known event stream incident events stream existed on the NPM registry since 2011, very long time practically did not receive any new releases in the last two years, which we'll see in a second, why this is so important, but it gained millions of downloads per week. Out of the blue, a person shows up on the GitHub repository, opening an issue and wanting to help as is customary in open source. They further contribute code with several pull requests, and then they also create a pull request that adds one of the pieces of code into a new dependency, a new module that gets added to event streams, own package dependency tree with supposedly genuine intent to improve the code base for event stream and use this in a modular way in the NPM ecosystem. Yet, if you could make Slater, they add a code payload inside that dependency that they own and they created.
Then they added now into event stream that extra piece of code that they added in the new version. Now inject smile or into a specific Bitcoin wallet application called copay. The copay wallet application used events three must part of its built process and is now tainted by the smile where this incident had gone unnoticed for almost three months, resulting with two versions released of the copay Bitcoin application that included the malware. Now, why was this important events room itself did not have any release in the last two years, an academic research paper published in 2019 investigated the properties of language-based software ecosystems. It compared the NPM ecosystem with the PI to pike and echo system. And it's found that 61% of open source packages on NPM could be considered abandoned because they did not receive any release. In the prior 12 months. The events, the event stream package would have been classified as an abandoned package in such research, just as much.
The thing is a year later as if we did not learn anything from the event stream incident, we have the electro native notify incident that took place. What happened? Well, let's go through this at one point in time, a user adds the electronics, not native, not if I package and it has no malware in it. Then the user adds it as a dependency to a popular package. The electron then the user releases a new version of that electron package, which now includes the malware. And the result is the Agama wallet. The gamma crypto wallet is now built with the most recent version of the app stream. Is it the X package now includes them malware. It sounds familiar exactly. It's the exact same thing that happened with adventuresome. And as if we did not learn anything from that, this is software supply chain security concerns that are all around us.
And the thing is those supplier and security concerns are not just open source related. We have witnessed it extend to the mobile application store ecosystem and sneak legend security research in the mobile application ecosystem. Identify the malware in the meantime, grow ad SDK them integral mobile SDK is used for advertising attribution for mobile applications and was downloaded more than 1.2 billion times a month. It is integrated into thousands of applications. Now these are downloads from all of us. This is you and me, all of us using the mobile applications, the app store to download the various applications from an official application store, not even open source. What sneak found interesting is that this SDK had a proprietary RP skated piece of code that's hooked into sensitive and careful system API APIs on mobile, which it shouldn't commonly do that. So these alerts or the potential suspicious behavior of what is going on with its Mintigo while at the SDK, what did it actually do?
One of the things that we observed is it intercepted all of those HTTP requests and it had this behavior where to avoid detection. The SDK would actually detect we're in debuggers and jailbroken devices or users then, you know, maybe hide itself. And lastly, sneaker sneaker observed that's. The SDK installed the back door used as a command and control instance into remote actors. So how much salt are we giving to the security of our own development infrastructure transcending from open source software supply chain, moving over from the mobile applications that are all using over to the development side of things that the, the tooling, the resources like cloud instances, you're staging environments, you're building continuous integration, tooling. How much of security mindfulness are we putting into those services that are used to build and ship all of that application code that we're building to our customers, to our end users.
So in January, 2021, not so long ago, as security researcher broke into Microsoft visual studio code GitHub repository, essentially providing him with the capabilities of making code modifications to the purple aura. Very well loved IDE that many developers use due to the problem of a command injection flaw that was made possible because of a flood regular expression. This allowed him to potentially open a new code, full requests, which upon it, the researcher was now able to execute codes that the vs code C ice scripts were running without requiring any sort of authentication or authorization checks, all of those things happening there in that open source repository publicly, anyone can see what is going on there. This led to remote reverse shells on the CEI servers. And from there, the ability to basically gain, push and right access to the repositories source code. Fortunately for us, the researcher reportedly responsibly reported this flood to Microsoft before advanced threat actors could actually exploit it yet.
Unfortunately for us, not all software supply chain attacks ended up in a responsible disclosure like this. Some of them hit us pretty hard and painfully, so was the case with April's 2021 just recent weeks ago of the disclosure of code Cubs supply chain security incident on April 15th, 2021, it was reported that an unauthorized access to a Google cloud storage key allow the malicious actor to alter the version of code calves up. Loader script code gov is a quality code quality tool used by many developers to assess code coverage and other code quality metrics. Now, what we can learn about the extent of this incident from the security week opening note article was that security response professionals are now scrambling to measure the fallout from a software supply chain compromise of code cuffs bash a floater that went undetected since January and exposed sensitive secrets like tokens and API keys and different kinds of credentials from different organizations around the world that use this in their CIO ecosystems for developers.
How did code cover learn about a security incident involving the malicious batch upload version on April 1st, a customer reported that the sharp one integrity signature used to sign the batch uploader was not matching that of a downloaded file four months later after this was happening. And the thing is how much do we know about the current state of open source security and what it entails in an effort to explore the security awareness of the Python and JavaScript open source communities. A group of researchers set out to investigate how maintainers work in the open source community with regards to their work, the ability to mitigate security vulnerabilities. One of the research questions was how quickly do you open source maintainers mitigate a newly published security vulnerability ones that have a CVE and now needs to be fixed. The research found that it takes about a hundred days on average for both JavaScript and Python maintainers to scramble away and start migrating or mitigating to a public vulnerability.
Is that fast enough for us as consumers of a library? Can we really ask more of open source maintainers who are voluntarily doing their best? Anyway, if we examine the number of commits that mitigate a vulnerability out of the total number of commits, we can see that Python has a more consistent track of security mitigation. Whereas the JavaScript community was largely inactive with this, with this regard until 28, 10, this demonstrates the low levels of I'd say obstacle awareness around the JS open source community in those early years, those early years before 2018 as a case study, we can refer to Mark's own security vulnerability from several years back marked is a markdown parser for the web. It is downloaded millions of times a week, one of the most popular libraries for this purpose of parsing markdown and using it in the server side or the backend JavaScript and node JS ecosystems.
But one day a security researcher opened a code pull request, which reports and fixes and XSS vulnerability across sites, 15 vulnerability that impacts marked on JavaScript obligations. The pull request included tests for future code regressions and proof of concept examples. So the maintainer would have the ability to reproduce the problem, but as is with open source, software maintainers are really just trying to do their best, but they can't be there all the time. And there are no legal or contracting obligations for them to support you or me or any of us. They're really just trying to do their best. So this vulnerability was lift out in the open with no fixed for a year, right? No fix, but everyone knows about this vulnerability. This security issue and proposed fix was open in 2015, but was only merged and made available as an official release in 2016 for a package that gets millions of downloads a week and powers who many notes, how many applications when we were also very much dependent on open source software, we can't ignore the question of where do we put our trust and what are our mitigations and security controls to cope with the risks involved in 2017, a security researcher working with the no JS foundation conducted the research in which he wanted to assess the state of weekend PM.
Credentials used by maintainers and developers pushing code into the NPM registry. When I take a long and deep breath here, because his work revealed the devastating truth of developer's lack of security hygiene. This security research was able to find how, and for myself, how this person was able to gain access, publish access to 14% of NPMs ecosystem system modules on the registry, some of these modules were downloaded tens of millions times a week. They were all powering and an essential key of this thriving JavaScript ecosystem. The thing is the problem was rooted with insecure passwords. They were chosen by the maintainers of these accounts, by the contributors who have access to publish them such as one, one packet. For example, one account, use the word password, literally the words, password for the account password. As the maintainer of a package that has millions of downloads for their package.
This is insane. This is unheard of. This is unthinkable. What could have happened if this person was not as security researcher working with the node foundation, but was the malicious actor. I'll let you ponder upon this and hope it doesn't happen. But if our code packages can reach thousands of millions of developers, shouldn't we have more protections in place. We're all citizens of this open source community. Some of us use open source software developers who build open source or contribute to open source software. We're all in this world together. Can we do better for our own accounts, security hygiene? So it's one thing. If we did not have support for more of this, you know, greatness of, uh, account hygiene, but we do the NP NPM, the largest registry of open source software packages spanning 1.5 million packages to date has supported the two factor authentication since the end of 2017 long time ago already, despite all of those security incidents and compromise accounts stories happening throughout the years in 2019 only 7.1% of NPM package maintainers have enabled two factor authentication, only 7.1%.
Did it go better? Unfortunately, not, not enough. The software supply chain security story is not resonating enough with developers because the year later in 2020 QFA enabled accounts have only grown by merely 2% to approximately 9% of developer accounts on NPR. People are not enabling two factor authentication for critical infrastructure for where the code lie for the software supply chain security that compromises all of us. If something goes bad, should we really be surprised by the fact that users are not enabling two factor authentication? Should we, should we be surprised that the big, bad passwords too? I don't know. Let's see, let's see. From 2020, uh, 12, when, uh, when LinkedIn suffered the massive data breach, in which one analysis of the data showed that 35% of 65 million accounts in the data set that leaked were accounts that reuse the password from a previous leak, there was a previous leak and they reuse the password, the same password, reusing it over and over again.
How bad are we talking about this, this problem of choosing passwords and managing them properly more than 750 thousands of accounts in that data breach simply had their user's passwords set to the string. 1, 2, 3, 4, 5, 6, almost a million of the accounts on LinkedIn. Use that as a password, almost two thousands of the accounts set their password to literally the word LinkedIn, to slog, to log in for their account. That's it, that's all you had to know to log in on their behalf. So what can we say more about this as cybersecurity experts? Bruce narrows said in his book secrets and lies, humans are often representing the weakest link in the chain. A term coin does Lena says low back in 1999, by every claimant in his work, they can take roll. And the beds are shifting us towards a different story and perspective of open source security and the open source ACA system.
In this book, in this article, he explored the differences between software development as executed within open source movement, this organized and that of enterprise as formally organized companies. And Eric stated that given a large enough number of developers and users of a software such as is common with open source communities, software floats will be quick to detect. And I ask you, is this always the case in January, 2021? It was discovered a pseudo at common utility installed in many of distributions has a security vulnerability existing for years, their studio application specifically any underprivileged user can now gain root access just based on the default pseudo configuration. What is so daunting about this vulnerability was that it was hiding in plain sight for a decade, for 10, whole years. It was just hiding there until someone found it. So we reached a point where we take open source for granted.
Open source registries are open in their nature and allow developers to openly push their packages to them. We have come accustomed to opening an issue in a project source code repository, asking for help asking for a feature. But what happens when maintainers pulled the rug off of our fits and start maintaining a library or worse, completely remove a library altogether from the registry. This is exactly what happened in 2016, when a maintainer pooled tenants of their open source packages from NPM. One of them was that pivotal package in the ecosystem and failing to download it now resulted in a widespread breakage of CEI and installed processes all over at the very least. This incident showed us two things. First, the weakness in how businesses failed to manage their open store software in a responsible way. And secondly, how registries did not even foresee this as a problem, they weren't, they're not designed to handle these kind of circumstances of maintainer pulling after libraries from a registry.
Why would they do that? So can we really deny being a part of an open source ecosystems as consumers, as contributors, maybe even as maintainers, we all play a part in a world where 90% of our applications code is made up of open source components. What kind of malicious activities and assets can we track back to open source echo systems, time after time, we find more and more malicious packages heating the NPM ecosystem. You may have been a victim to one of these. If you may be misspelled the package known as a type of squatting at dock, when you try to install it, or perhaps someone Flint at one of those malicious package packages in a dependency tree that your application is dependent on and malicious packages. Aren't just a thing on the JavaScript ecosystem on March, 2020, one more than 3000 malicious packages were published in bulk on the registry.
Now to further show us how attackers can harness this open source ecosystem and registries and package managers into their own advantage. Alex, Pershant published his research in February, 2021 about how he exploited design flaws in package managers, registries, and human error to infiltrate into corporations such as apple, Microsoft, and others. So I would like to leave you off with the following questions to ponder upon. Are we going to have less or more software in the future? Are we going to use less or more open source software specifically? And who do you trust? I am Lauren tele developer advocate at sneak a where we build a security platform to help developers build securely with the open source software. Thank you all for joining my doc.