Exemplars, Laggards, and Hoarders: A Data-Driven Look at Open Source Software Supply Chains

In a year long collaboration with Gene Kim and Dr. Stephen Magill, we objectively examined and empirically documented software release patterns and cybersecurity hygiene practices across 54,000 commercial development teams and open source projects.


In this session, we will present evidence on the outcomes of that research, highlighting organizational and technology practices that enable exemplar open source teams to deliver 50% more commits, release new code 2.4x faster, and remediate security vulnerabilities 2.9x faster, all while delivering a level of value that makes them standouts in terms of popularity and adoption.

GK

Gene Kim

Founder and Author, IT Revolution

DS

Dr. Stephen Magill

CEO, MuseDev

Transcript

00:00:08

Hi, I'm Steven Miguel. I've been doing academic research and software analysis, security and programming languages for more than 15 years versus part of my PhD work at Carnegie Mellon. And then at other universities and industry research labs over the last few years, I've been getting more and more interested in the practice of software. I in particular open source development practices, how enterprises approach software and how to best contribute to these communities by improving tools and practices and how this all comes together to generate modern software.

00:00:39

And, uh, uh, I, my name is Jane Kim. I've been studying high performing technology organizations for 20 years and, uh, one of the funniest things I've ever gotten to work on in my career was something called the state of DevOps report with Dr. Nicole Forsgren and, uh, judge humble, which resulted in the accelerate book. And, uh, it was so fun to be able to take me of the things I learned there and apply it to a separate project. What I will describe in just a moment. So this is a reprise of a presentation that, uh, Dr. Steven McGill and I did, I could help universe last year, and I thought it was so relevant to the technology leadership community that I asked Steven, if he'd be willing to present this again with me, uh, here at DevOps enterprise. So in terms of their problem statement, uh, back at GitHub universe and that freedom and the CEO of GitHub said that 99% of all new software projects use open source in some fashion.

00:01:26

And then Erica brushes, our CEO said, uh, we are inviting thousands of developers into your code. Uh, when you use open source dependencies into your living room and kind of Steve and I were in the audience, and we were laughing going, is that a good thing or a bad thing, right? Then as I got trashed the place, are they going to help with the kitchen project? And so that's actually what we want to understand better, uh, what practices, so, um, in result in good security outcomes in the, uh, software supply chain. So on the next slide, uh, this is just a brief thumbnail of the state of DevOps research over six years, we've benchmarked across, uh, over 36,000 respondents, really trying to understand what does high performance look like and what are the architectural practices, technical practices and cultural norms that result in great performance.

00:02:12

And, uh, the, what the punchline is that we found this, these group of it performance metrics, uh, that were dominant deployment frequency to lead time, uh, change success rate, and meantime crew store. And so the goal of this study is really to understand what structures and practices correlate with exemplary outcomes, uh, such as fast time to update fast hunter media security vulnerabilities. And the question was, will we find a lot of the practices that we found in the state of DevOps report also applicable in open source projects? And so the amazing opportunity that came up was, uh, when a couple of years ago, some friends at phenotype reached out to me with what I thought was an amazing opportunity, which was to be able to analyze the data in a Maven central, which struck me as an incredible opportunity. So for those of you who don't know Maven central to Java as what NPM is, the JavaScript, uh, PI is a Python and gems is to Ruby. And so it is the second large largest package repository in the world. And this was especially interesting to me because my favorite programming language, uh, is closure. And, uh, it leverages all the amazing Java components, uh, inside of Maven central. So it just struck me as something, I got a tremendous joy out of doing, and it was amazing to work with the team of Sonatype. And, uh, it was also delighted me. The Steven has a particular fondness for Maven as well.

00:03:33

Yes, yes. You know, I mean, Haskell is very my, my first true programming language, but a Scala as a statically typed, uh, function that mostly functional language, uh, you know, it's definitely close to my heart and being able to leverage, uh, everything in the Java ecosystem. Certainly amazing. So, you know, it's, it, it's a great set of components, a great dataset to dig into from an analysis perspective.

00:03:54

Absolutely. And, and so this is what resulted in, uh, collaborating with, uh, Sonatype on their state of the software supply chain. And so we worked with, uh, not just, uh, Dr. Steven McGill about Bruce Mayhew and engineering director there at Gazi Mamood, uh, and Brian Fox, who was actually one of the inventors of Maven and Maven central, uh, which was, uh, uh, treat beyond words. So, uh, so it's really the hypothesis that we want to focus in on for these hot spots is one.

00:04:23

Yeah. Sorry. Yeah. So, yeah, so we had a number of, uh, things that we wanted to test, right. Um, we wanted to see if projects that release frequently have better outcomes, right. Because it's something that we see, uh, in enterprise, how enterprises operate do we see at an open source, uh, do projects that update dependencies more frequently? Are they generally more secure, uh, do projects with fewer dependencies stay more up to date? Right. It seems like it should be easier to stay on top of things when you have fewer things to keep up with, and then our more popular projects better about staying up to date, right? These are all sort of intuitive outcomes that you would expect given our experience, our day-to-day experience, developing software, but do they actually hold when you look at the data, you know, does it back these up?

00:05:04

And yeah. And so we had the opportunity to investigate this by digging into this Maven central dataset, which, um, when I pulled these numbers, uh, it was up to 310,000, over 310,000 Java components. Um, those are there's of course, multiple versions of each component. So when you look at the actual releases, you're over 4 million different releases and a whole lot of those are associated with GitHub repos, which just gives us another data source to draw on. Right. We can go in and look at that metadata from GitHub to learn more about those projects. Um, you know, and, uh, amazingly, you know, almost 9% of those have known vulnerabilities, you know, their, their vulnerabilities known against those components. And so we dug into that vulnerability data as well. And, uh, and in particular, what we focused on here was the dependency structure, uh, of the components and how do they treat their dependencies?

00:05:55

Um, how often are they upgraded when there's someone else's dependency, right. And, um, this dependency structure is really complex and rich, right? When you think about all the libraries that you're pulling in and all the libraries that they're pulling in transitively and so on, you know, it's this huge collection of things that, you know, your application code becomes this small, tiny little bit of the system. Right. Um, and so we thought it was really valuable to look at this, the security posture of the rest of that system. Right. What do you see, um, in all those dependencies and transitive dependencies, and what should that tell you in terms of how you should treat, uh, the dependencies that you bring in and, and things you should consider, uh, there? So, yeah, if you're, if you're way down here, uh, you know, it's a small part of this ecosystem. So, uh, to look at this, we, we started with, we started with all components published in, in Maven central. Uh, we focused first on, uh, based on time. So we looked at the last five years, uh, because, you know, practices change over time development trends, change over time. We wanted what we discovered to be relevant for software development as it's done today. Right. And then, uh, then we looked at components that are actually part of that software supply chain. So if you look at dependencies,

00:07:02

Want to advance the slides, Steven,

00:07:05

Uh, yes. Yeah. Uh, yeah. So if you look at the dependency structures, you know, they are connected into that, that tree of dependencies, right. They're used by someone or they're using some open source components. Um, and then we looked at, uh, components that follow Maven central guidelines for versioning, right. So, um, we have to be able to tell when a component is, is actually upgrading and what, what are major releases and minor releases. Um, so having correct versioning posture was important for doing the analysis. Um, and then, uh, you know, we, like I said, we want to look at the dependency structure and transitive dependencies. And so we required, you know, at least all the dependencies of a project to also satisfy all these, right? So you do all that filtering, um, to get really a good data set that you can analyze in a variety of different ways. Uh, and that leaves us with 13.6% of, of those components that we started with. And so these were really the components that they're part of the supply chain, they sort of following best practices. So then we can dig into, you know, what differentiates them from a security perspective.

00:08:05

And we looked at, um, a number of attributes here. Um, so across this dataset, uh, we collected information on popularity, which we measured as average daily central repository downloads. How often are these polls, uh, by users of that library? Uh, how frequently do they release, uh, average number of commits per month size of team presence of CIA? Those three are all things we actually get from that connection to get hub metadata that I mentioned earlier, um, we also looked at foundation support, right? Is there an open source foundation or, uh, that is providing, you know, providing resources to these, uh, these projects. Uh, and then we looked at security, uh, data as well as, um, this update lag. So how they treat their dependencies.

00:08:47

Yeah. So, um, one of the, uh, areas of data was those that we could actually grab from the repository, they can merge with our dataset. And so I'll go through kind of the major ones. Uh, so the popularity metrics included as Steven mentioned, uh, the download number of daily downloads from Maven central. We also lose this, use this amazing data set called libraries at IO. And so we could actually, uh, it's amazing for each repo. We could actually look up the number of GitHub stars, forks, and pull requests, right? So that's the sort of a proxy for popularity as well. And then we use an internal, um, Sonatype got access to Sonatype enterprise data in terms of how frequently certain repositories were, uh, updating, um, on, uh, uh, through their nexus IQ product. We also, uh, were able to go into the GitHub repos and look at commit activities.

00:09:33

And so a written in my favorite programming language closure, it was so fun to be able to go and get the, uh, number of commits per month. Uh, and then also look at the measure of the number of developers, uh, on those teams. So we would just as measured by number of developers who would commit in a given month, um, sorry for every month measure, how many developers are committing, and then we can, uh, compute what the average, uh, uh, size of the developers, uh, too many developers are. And then, uh, we also looked for presence of a CEI on those GitHub repos. So any trace of, uh, Travis configuration file, Jenkins, CircleCI, and so forth. And, uh, that'd be the last time I mentioned the CIO presence. I mean, it just stands to reason that, uh, you know, repos that had convinced integration would perform better than those who didn't. Uh, but that was not the case. There was actually no difference in any metric, uh, uh, those who had CIA and those who didn't, which was very surprising to me is that's the first of many big surprises in this project. And so that's the last time I mentioned that,

00:10:32

But, but what did perform well wear jeans closure skills? So the efficiency advantages of closure are real, uh, that script came together very quickly.

00:10:41

Um, so we also gathered, uh, project level data looking at, um, as I mentioned, the support, uh, that, uh, that our project might have, is it foundation supported? Does it have, uh, those extra resources that can really help make a difference in terms of, uh, uh, how, how much effort they can put into development? Uh, we looked at a number of dependencies, which we took as the maximum count of dependencies for any components. Um, we looked at sales dependencies and I'll say more about what a sale dependency actually is a bit later. Um, and then this release period. So the frequency metric as a key metric here, how quickly and frequently it does a project release.

00:11:17

All right. Last thing in terms of background. So I'd mentioned the key performance measures of, uh, from the state of DevOps report. And so one of the things that we did was just kind of explore what the parallels would be between this work and Steve DevOps, uh, research. So deployment frequency, you know, we kind of said, all right, this sort of analogous measures are commits per month releases per month and commits per dev per month, uh, deployment lead time. You know, we wanted to look at, you know, PR uh, lead time than in other words, how long did it take for a PR to actually get closed, uh, issue resolution time, uh, deployment success rate, right? And we'll kind of map that to, you know, uh, breaking changes when you update it. Dependency directors have that break functionality, uh, meantime to restore, to meantime for immediate security volumes, meantime to update when new components become available and, uh, the stale dependency, uh, profiles, and then organization forms might map to popularity stars and so forth. So we didn't get a chance to go through all of these, but it did kind of give us a mental model of how do we serve, uh, use and map the state of DevOps research and thinking, uh, to this project.

00:12:17

Oh, so hypothesis number one. Uh, it was exactly that. So one of the decisive findings in the state of dev ops research is that those organizations that deploy more frequently because they have better deployment lead times, you know, definitely have better performance. So the first question is, do we find the same similar findings, uh, in this dataset as well? And the answer is Steven,

00:12:37

The answer is yes. So, um, probably very much so, so projects that released frequently, um, have better outcomes. And, um, what we mean by better outcomes is, uh, they're more popular. So, you know, the, in terms of organizational performance metrics, right, you're contributing to open source, so other people can use it, you can make the world better. So what better impact look measure of impact than popularity? Right. Um, so they're five times more popular on average. Um, they also have a larger development teams, uh, and we didn't, uh, you know, we didn't get into, uh, the causal connection between these things. These are correlations that we're talking about here. Uh, so we don't know if it's, uh, you know, they're releasing more frequently. So developers see that their code changes get in and get out there. And so that attracts more developers or the larger team itself is what lets them release more frequently, but those are definitely connected.

00:13:26

And then, um, they also in general have a higher foundation support rates. So, um, the next thing we looked at was the security side of the equation. So those are, those are outcomes in terms of usage, popularity. Um, but what about security? Uh, and so here, uh, we collected a number of security, relevant metrics. Um, and so just to give you like an image of what this looks like here is, uh, the dependence, a sample dependency structure, right. For some made-up components, right? So we have a, B and C, you know, C depends on DNA, uh, and there are several version releases of each of these components. And so if you think about a vulnerability coming out in component B, uh, in that red hexagon there in the middle, uh, that vulnerability happens at some point in time. Um, at some later point in time, B releases a new version, uh, that corrects that vulnerability. So we're seeing B version 2.3 fixes that security vulnerability. And then at some point later, see,

00:14:23

Build ahead, Steven.

00:14:25

Yeah, thank you. Um, at some point later, C adopts that version of B, right? And so, um, there's this period of time where it be as vulnerable. Um, and because C uses B there's some period of time where, uh, C is vulnerable, uh, but really, uh, C as enabled to remediate this vulnerability until B does their job and fixes, fixes the security vulnerability. Uh, and so what we count as the remediation time is really that period, right? So when does the fixed version of BB come available and how long does it take see to then apply that fixed version? So we call that, uh, the remediation time or the update time. Um, we, we use the term update time to refer to any update, right? So like if, uh, C as adopting this new version of a, um, that counts as an update, even though a, in this example, it doesn't have a security vulnerability.

00:15:13

Um, and so, so the terms to keep in mind are, uh, remediation time and update time where remediation time refers just to those updates that are security relevant. Uh, we also have this concept of a sale dependency. So, um, it's often the case that, uh, some version comes out, but, um, uh, it's never adopted by C right? So C uh, you know, C continuous use the old version of a, and so we consider this, uh, dependency to be stale at that point. So, so these key metrics that we track our time to remediate time to update and sale dependencies, uh, we look at the median of these actually, because, uh, these data sets are log normal distributed. And so, um, again makes the most sense there. Um, and, uh, this is the data that we found,

00:15:59

Oh, yeah, it's not good. So, um, so this is a visualization of what the update behaviors are. Uh, and so if you look on the X assess time, and so the clock begins whenever there is a, uh, vulnerability detected and the Y axis is, uh, at any given point in time, what percentage of the population on average, uh, has updated? And so the way to read this, um, appallingly bad profile is this, the median time to remediate vulnerabilities, um, is 180 days. So that's like a half year, right. Uh, the mean is even worse. There's almost a full year now. And, and so how long does it take for 95% of the population to update is 3.5 years. And so we actually had to cut the graph off at three years because I likely wouldn't even be able to see the curve. Um, and so, so that is, I'll just take a moment to say that, to say, you know, that's a much worse than I thought.

00:16:50

Um, and then, uh, so, uh, here's another way to look at it is, uh, you know, let's take a look at the bottom of 10 corner. Why do who's updating quickly? Um, and you know, what, what factors cause that? So one of the things that, uh, was a problem, uh, is that it's actually kind of a sparse dataset. Uh, and so we actually looked at update behaviors, uh, and it fits a very similar curve. Um, uh, it takes a little bit less time. The median is a hundred, um, 130 days. Um, uh, but you can see that there's a very similar profile, uh, between MTU, meantime, to update for us the meantime for immediate security vulnerabilities. And, uh, my friend Steven says, oh yeah, they both have a log normal distribution. Um, so which gets to the next point.

00:17:33

Yeah. So, um, right. So there's, um, uh, you know, there's clearly a population that a population level, the distributions line up here, um, in terms of, you know, how quickly it takes to, uh, for a certain percentage of the components to update or remediate. Um, if we also look at, for an components, um, there, uh, median time to update versus remediate, um, that matches up very closely. So this is on the X axis. We have a time to update, um, for non-security relevant updates on the Y axis, it's time to update for security relevant updates. And we can see, um, that, uh, you know, there is a correlation here it's not perfect, right. Um, but, uh, you know, 0.6 Pearson correlation coefficient here. And so, um, by and large, uh, projects that update more frequently tend to be more secure and vice versa. Um, so in fact, if you dig in a little bit deeper, um, 55% of the projects have an MTTR and an MTT you that are within 20% of each other, uh, and only 15% of the projects, uh, that have worse than average, MTT you managed to maintain better than average MTTR.

00:18:37

So this gets to this question we had of like, uh, you know, what about that mythical project that like, they only update when there's a security vulnerability. Right. You know, they say, you know, they keep, keep things that, but they're really paying attention to security. Like there just are not a lot of those projects. Um, and so, yeah, so most, most of the projects, uh, you know, fit within this cone. That's, that's close to, uh, close to, you know, uh, performing the same on a security and non-security relevant update basis. All right. So that was hypothesis as to actually that, um, updating more frequently, uh, it corresponds with a higher security level. And, uh, that was validated. Yeah. So, you know, sort of all of that data points to, uh, this fact that, that security and update behavior tend to go hand isn't it.

00:19:22

So that actually mirrors something that Jeremy Long and the inventor of the wasp dependency check project said, this is the best way to stay secure is just to update your dependencies. And he said, uh, you know, uh, that, so many of the, uh, he had a great example of there was the open faces project. They had a vulnerability that was actually fixed in 2006, but the first CV was disclosed in 2018. And that's when all the Bitcoin miners took over these vulnerable components. So it was actually the publishing of the CVE that actually led to the exploits. And so had they just stay current, they would not have had any problems. So again, it was this points to saying the best way to secure, just integrate updated dependencies into developers, daily work

00:19:59

That's right. Yeah. That, that version you're upgrading from might not have a known dependency right now, but that doesn't mean one won't be discovered later. And so if you just update, then you're protected. Um, so hypothesis three was that products with fewer dependencies will stay more up to date. You know, it's sort of easier to manage a less complicated mess. Um, surprisingly this is not the case. Uh, and it wasn't just that there was no connection between the two. It was actually the opposite connection of what we expected. So components with more dependencies actually have a better view on average and it's statistically significant. Um, so that's weird, right? So we dug into that, um, and discovered that actually there's another factor at play here. So projects with more dependencies also tend to have larger development teams. Um, and so, you know, this may be explains part of why having those extra dependencies, you know, it can be managed, right. They can, they can still manage to stay up to date. Uh, cause they do have more developers contributing here. Um, and so, you know, this is we, we grabbed it, right? Here's the connection on the X axis number of dependencies, why exercise the team, you can see the team growing, uh, and you know, uh, moving to the point where you have to buy an extra pizza.

00:21:15

All right. Um, so hypothesis for, was that more popular? Uh,

00:21:21

Oh, so actually to me, just a little commentary, right? So this is, I got an echo Steven here, right? This is so surprising, right. That more dependencies mean they're actually better up-to-date. And so, uh, w which way what's the causality, is it that having more developers, um, causes more dependencies or do the number of dependencies, Chris' much worse that you have to get more developers and it's just a super interesting. And, uh, yeah, so let's just say that's an open question. Very surprising.

00:21:46

That's all right. Is everyone bringing their favorite unit testing library and you end up with five different ways of testing or, you know, what's going on there definitely is something to dig into in the future. Right. You want to talk about popularity genes?

00:21:57

Oh yeah. Well, um, sure. So, um, that hypothesis four is more popular projects will be better about staying up to date. And maybe just to set the context of this is that, um, you know, whenever you have a problem to solve, you have, uh, logs you want to generate, or you want to aggregate a statistical, you know, I use a statistical package in general, right. Uh, you kind of pick the most popular project as measured by the number of stars or forks or something. Right. I mean, that's the way I do it. So it sure stands to reason, uh, that popularity would be a good, um, proxy for how much you can trust the software component. Makes sense. Yeah.

00:22:32

I agree. It seems intuitive. So let's look at the data.

00:22:36

Yeah.

00:22:38

And what we find is that, uh, actually not all popular projects are exemplary and release fast and stay secure. And up-to-date um, so like, uh, in, in red here is a, those projects, uh, that do not have good update hygiene. They're not keeping their dependencies up to date. Um, on the Y axis is popularity on a log scale. So things get much more popular very quickly. Um, and you can see, you know, what's in this box here is, um, this really large group of, uh, very popular projects that are lagging behind when it comes to update behavior.

00:23:12

Yeah. Uh, so refuted, I guess, right. Just Steven.

00:23:18

Yeah. There were plenty of popular components with foreign TTU, uh, and even more so popularity does not correlate with MTU. So, um, the most popular projects are not statistically different from any others with respect to MTTR.

00:23:32

Yeah. So, I mean, this is kind of a, um, a problem for me because it says this Horistic, I've been using. In fact, I just used it a week ago, right above, like, how do I go when I go shopping for components to use, to solve my problem? Uh, I don't have a lot of great alternatives and, uh, besides looking for the number of stars and forks, and I said, well, let's put that chalk that up for future work.

00:23:51

Yeah. All right. So then, um, we did some clustering, uh, based on this. Uh, what can we see if we, if we look at this dataset and we try to break it apart into, you know, which, um, projects are close to each other, which are farther apart in terms of their update behavior and how similar that is, you know, how does this break down? And we basically see, uh, five different clusters here. Um, there are two exemplary clusters that are very, uh, very performing very well in terms of update behavior. Um, and we divided those further into small teams versus large teams. So, um, you have large teams that are really keeping up to date performing super well, um, average of nine, uh, developers per team for the large ones. And then you have a large group of these really small, you know, single dev two dev projects that are also doing a great job keeping up to date.

00:24:40

So, you know, it doesn't take, you don't have to have a large development team to stay up to date with us. Um, those are very different behavioral lead from, uh, what we're calling the laggerts, right? These are the, these are the projects that do not have great update hygiene they're in the bottom, uh, 20%, uh, in terms of update behavior. Um, and then there's two really interesting, uh, classes that are differentiated in terms of, you know, sort of quirks about their behavior when we're calling features first. So they, they actually released frequently. They have, if you will, a lot of release bandwidth, right. But they're not using it to stay up to date. Um, they're just, you know, maybe adding new features or improving performance. There's something they're not keeping on top of, uh, their dependencies and updating those. Um, and then we have what we're calling the cautious crew, right. Which is, uh, they, they actually do sort of maintain a cadence of update. Uh, but there are like a version or two behind, right. So they, they sort of wait, they don't want to be that first mover. Right. They wait for other people to take this stuff.

00:25:34

Yeah. My favorite one is a feature. First one. It's like they, they seem to have the technical practices to do high rates of releases, but they don't care about updating the dependencies. So I think those are promising cluster because they can do the, probably easiest for them to get, become an exemplar. The other interesting colostomy was a what in my head, I call it the open source industrial complex, these projects, the large exemplars, large teams with 10 active developers, or more constantly committing, uh, and it's probably part of their day jobs. So, uh, what's interesting to me is that the examples are not the domains of these OSS industrial complex. And it can be found examples can be found both in large groups and small groups as Steven said.

00:26:13

Uh, oh yeah. So, uh, Jen, you want to talk about the survey data that we collected as well? Yeah,

00:26:17

Yeah, yeah. Also this was really fun. Uh, so, uh, it was actually, we, we saw this and we wanted to actually put out a survey, uh, and with very short notice, just to see if we could test some hypotheses in terms of the behavioral, um, how, how did they think about it and what are the practice that led to these great behaviors? So we're able to push out a survey that went out to 658 respondents. And, uh, what was amazing is that we were able to cluster them into high, medium, and low amounts of pain, uh, associated with our updates. And so when you compare the high pain versus low pain, updating dependencies is three, a one-third is when you compare high paint, a low pain they're three times more likely likely to say update dependence is painful, update, vulnerable components, two foot, and a half times more likely to say as painful.

00:27:03

They are 10 times more likely to schedule, update dependencies as a part of their daily work. Uh, let me just, that's amazing. Now we still strive to use, uh, the latest version. So it's the six and a half times more likely to have the 11 times more likely to have some sort of process to add, remove, or, uh, dependencies, I'm sorry. Uh, add dependencies and 10 times more likely to have a process of proactively remove problematic dependencies and to have 12 times more likely to have automated tools, to monitor policy compliance around dependency. So like when you see a difference like that, uh, it just really says there's something really there that says it, behaviors are different that leads to these, uh, different outcomes.

00:27:41

Yeah. And you know, what we see here, I think really is that the practices that we expect to go together and to support, uh, staying up-to-date and staying secure, um, you know, we are at least in this self-reported survey data, uh, you know, seeing those tracks together, seeing those come together. And, um, we're doing actually a lot of research right now and, and digging through the data on, uh, this year's version of the report. And the one thing that we're focusing on is, uh, this side of the equation that's usage side and what people are discovering there. So, um, yeah, so, you know, uh, really excited about all the work that we were able to do together with the Sonatype team, uh, that fed into this report. Um, if you're interested in seeing the report, you can go find it on the sort of type website, or just email me that address.

00:28:26

I'm happy to just send you a direct link to download it. Uh, and, you know, as I said, uh, we're working on the next iteration right now and, uh, studying that consumer side. So we looked a lot at the open-source producers in, in what we're reporting here. Uh, now let's dig even more into, uh, the consumers of that open source software and in particular, look at their practices and the impact that has on various outcomes. So in the future, we want to also start looking at breaking changes, you know, like, uh, what is it really look at the connections between the consumer side and the producer side. So what practices, when it comes to open source library authors, what practices of theirs can help the consumers, uh, more effectively use those libraries and more effectively stay up to date. And so breaking changes might be one of those other might be other practices that also really help, uh, that whole supply chain work smoothly.

00:29:17

Perfect. And if you go to slide help, we're looking for we'll post this in the slack channel that they're specifically looking for help for future research in terms of how to detect breaking changes, uh, is immutability of API APIs that help, uh, get data and pull request lead time and issue resolution time, and find some sort of authoritative service source for foundation support. Uh, so we'll post that in the slack channel and, uh, Steven take it away.

00:29:40

Yeah, that's right. So, you know, quick takeaways, what practices can you adopt based on this, you know, integrate updating dependencies into your daily work, right. Uh, it's not about picking and choosing, just always be up to date, um, contribute updates back to components you use, uh, don't make decisions based just on popularity and, uh, you know, and tell us what hypotheses you're interested in. Um, cause like I said, we're continuing to do data analysis. Yeah.

00:30:03

And, uh, Steven mentioned that we were deep in the midst of analyzing data for year two of this collaboration and it's super exciting. So, uh, stay tuned for that.