00:00:02

Got to be here for the fifth time. Uh, started speaking at this conference, uh, back in 2014. And over the years through the transformation, I have lost a lot of my hair. So, uh, my name is Topo Pal. Uh, I'm senior director and senior distinguished engineer at Capital One. Uh, basically I'm a developer, uh, DevOps evangelist. I'm also a member of our open source steering committee, and I'm the creator and core contributor of our, uh, flagship DevOps dashboard project, open source called hy.

00:00:39

Yeah. And for those I haven't had the pleasure of meeting, my name is Jamie Specter and I am very, very excited to join Topo today on stage to talk to y'all. Um, part of my role at Capital One is open source council, and what that means is I get to partner with topo, I get to partner with our open source program. I get to partner with our development community on all things open source, and that includes use contributions and external projects. You know, I've been with Capital One now for six years, um, and I've had the incredible opportunity to partner with, um, people all across the enterprise. And that's one thing I love about this conference so far. This is my first time here. Um, is that consistent theme related to just the value of partnering with others? I've seen a lot of value of that come from that, and you'll hear that theme actually today in our, in our presentation.

00:01:26

So, let's see. So many of you may know, capital One is a credit card company. We're actually one of the largest credit card companies in the US with over 70 million accounts. And you may also know that Capital One is also one of the largest banks in the us. But what you may not know and be aware of is that Capital One is a founder led technology company. In fact, um, we're about to celebrate our 25th anniversary of from going public next year. Our youngest competitor is actually 108 years old. So in that sense, we are a startup. And, um, because we're founder led, we have that entrepreneurial mindset. You know, from the day we were founded, capital One has always tried to be, uh, disruptive. It's part of our DNA and um, open source. And DevOps has contributed to that success despite the regulatory regulated industry in which we, um, operate.

00:02:20

We are different. We have a different DNA because we build our own software. We build on public cloud, we deploy our software on public cloud, uh, using microservices architecture, and we use a lot of open source. Uh, we are open source first company, and we strongly believe in continuous delivery. Uh, especially in the regulated industry. Our belief is that if you are not doing continuous delivery in a regulatory industry, you are not, quote unquote, compliant enough. About six years ago, we were mostly outsourced a hundred percent waterfall manual processes all over the place. We are slow, as you can imagine, and only commercial softwares are allowed. And we also said no to open source. That was like six years ago. And then six years of journey to today, it took us from mostly outsource to mostly insource. And I think this is very important for, uh, a digital transformation, vertical silos like ops, Dave, qa, blah, blah, blah, to a product centric approach where the product team includes everybody that needs to be there to actually do everything related to that product from super nuts.

00:03:33

And as I said, from Dave Ops, qa, rm, we had these different roles. Your developers, your operations, and all that to we are all engineers, developers write some code operations, write some code infrastructure as code qa, engineers write some code testing and all that. Even release managers have become release engineers. And we spoke about that in, uh, 2014. I talked about building out our automation steps. In 15, I talked about how we tried to scale DevOps and open source and cloud innovation across the enterprise. In 2016, we talked about how we are concentrating on measuring, improving, and maturing our developed success. And then last year I talked about better governance via continuous delivery. And that is very core to us because we are regulated. Uh, we cannot do things that are not allowed. So we had to be very careful about our, our, our journey, and we decided, and we believe that continuous delivery is the way to go. So this year when I was thinking of the topic that I'm going to talk about, uh, I found, uh, state of DevOps report from Dora accelerated state of DevOps for the first time. They have included open source in the study. And I thought that is, uh, uh, a kind of natural, uh, progression, uh, to talk about open source and, and, and DevOps. So that's the thing that we are gonna talk about. As I said, about six years ago, we said no to open source.

00:05:07

And when you say back in the days, say no to open source, I also talked about many enterprises like us, larger and smaller. They all say that say no to open source, but use this code in Java, use your spring framework code in Eclipse, IDE use Apache, and, but say no to open source. I don't know if that makes sense or not, but what it means is that everybody used open source, but they are in a denial mode or something. Now, back to 2012, uh, when we first started our continuous integration journey, we started with this basic tool set on commercial tool. And then I brought in Hudson, because Jenkins was just born that time, and there's a lot of things going around into Jenkins and Hudson. We decided to stick with Hudson. We brought in Maven, sonar, cube Nexus, and then it blew up.

00:06:07

It didn't work. And that is because the commercial source code software, uh, didn't have enough capability rather to kick Hudson build jobs. And we went back to the vendor, vendor said, oh, the version that you have doesn't support that. You need an upgrade and that costs money. And we kind of said, okay, we are not doing that. We are bringing it a version. And again, today we don't use a version, we are on GitHub. But back in the days we are, we started using servers. And on day one it worked magic. But that was a big deal from commercial software to commercial source control to subversion, because free tool, are you going to use a free tool in our critical business needs, such as the thing that holds all of our software that's risky. What if it breaks, uh, uh, who to call? I don't know.

00:07:01

Uh, give our IP away because we'll be putting source code in that open source code. Does that mean that we have to open source our code too? Uh, what if somebody steals our code, meaning that some hackers put something in subversion that sends every commit to some cloud version or somewhere in the world? Uh, so we went through all these problems and we kind of talked about this. So finally we came to a realization that to, to get subversion in our organization, we at least have to buy subscription to some support. And we did that for the first year. We paid less than 1% of what I would've paid to actually upgrade my commercial software. And guess what? In one year, we never had the opportunity to call anybody because nothing broke. So we moved from commercial, uh, software to commercial SCM software to server version.

00:07:54

The other big thing was Maven we used and, and everybody had their libraries in their source control and they had their and scripts. Uh, it took days to even stand up a project in individual developer's, laptop, and then building was a nightmare too. So we decided we are going to use Maven. We are gonna put Nexus and put all the libraries in there, and we are going to resolve, uh, all the dependencies to our CI pipeline against, uh, nexus and in-house, Maven Central, if you will. Now, that posed another problem. How do you get the libraries? So before coming to Maven, we had this as our way to get libraries. So this is a open source, uh, acceptance form. You fill this out. We first print, print it out, and then type your name, your title, uh, date, and identify the open source software. Each and every library needs to have one of these. Actually, there are multiple pages. I just printed the first page, uh, just for demonstration purposes. Uh, if you, uh, and then there are some critical, uh, important questions asked from you. Are you going to modify this software? Yes or no? And if you said, yes, you are in trouble, uh, <laugh>.

00:09:10

And then, uh, do we intend to distribute this software along with, uh, your source code outside of Capital One and all that? And we printed that form, got it signed by your immediate manager, your VP and all that. And we did this. So that's the facts going. And once the facts went there, the legal department looked at it and then they said something. But the question is, why is facts <laugh>?

00:09:46

What is facts? Because back in the days, and I think it's even now, uh, our legal department or, or in a part of a building, which is kind of secluded, uh, there's a huge glass door and you needed special batch to even get in there. And that's the first reason. Second reason is you could not reach out to the legal department directly. You couldn't call them, uh, your email were not answered because in fact, they're busy in something else. So once I was walking past that building, I saw one of my colleagues standing in front of the glass door and knocking on the doors, and he was carrying a big bunch of printed sheets. I said, what's going on here? Why are you here? He said, I have these open source libraries approval to be obtained before I go to release, and my release is coming tomorrow. I said, why do you need the approval now? He said, if I don't get approval for these, the cab change approval board won't approve my release. And he was kind of knocking on the door. So he finally got approval. I mean, just by looking at this piece of papers, somebody would've approved it anyway, so, uh, <laugh>

00:10:50

<laugh>.

00:10:51

So then he and I decided that this doesn't need to be this way. There has to be a better way. No engineer should go through this. And we decided that we'll set up meetings with our legal department to talk about open source approval process. And we learned a very important thing. First of all, they didn't know why we are going to them to get approval for open source, and we didn't know what the risks are, and we are kind of never talked to each other. And that's why we kind of sat together and understood what they mean by approval, what we need software open source software for. And that whole realization came to, uh, uh, a kind of success, which Jamie's going to talk about our open source in tech process and all the risks.

00:11:38

Awesome. So first, let's think about like why we're here. What is the DevOps Enterprise Summit? And the goal of this conference, right, is just to give the leaders the tools and practices that they need to develop and deploy software faster and to win in the marketplace. And so what I'm here today to do is to provide you that information, to have this discussions about using open source. But what I'm not here to do is provide you specific tools and processes to implement in your companies. You are the expert for that. You know what works best for you, but equipped with the right knowledge, you can actually take advantage of the value that open source brings. So let's see, something for this group to think about. So we take a lot, um, we take great care in writing the code for our company. We do extensive due diligence in procuring our commercial software.

00:12:25

So why wouldn't we u uh, treat our open source that we use in the same way? Um, we want our applications to be healthy. And so we wanna make sure that we're protecting our companies no matter what we're bringing into our environments. So first, we need to understand how much open source we're using and being in Vegas. Um, I'd like to make a bet with everyone here, and I bet that you are all using open source. Um, now I'm not usually a huge gambler unless I have some kind of inside knowledge. And thanks to Black Duck and also, um, the doers report that Topo mentioned earlier, I have a little bit of inside knowledge that I'll actually share with you. This data right here is based on a black duck survey of over a thousand code bases and, um, every industry you can think of.

00:13:10

And what they found is that 96% of those applications contained open source. And of those 96% applications that contained open source, 57% of the code base was actually open source. Um, let's see. And this visual here will drive it home. And what this drives home is that yes, you can think about going out to a repository, downloading these soft, uh, the open source and bringing it in, but open source is actually coming into your environment from a number of different avenues. So the takeaway from these two slides is one, basically all applications contain open source, and more than likely, on average, your applications are containing more open source than even your proprietary code. So it's really important to understand, um, the risk. But before we get there, let's kind of talk, think about how we think about open source. So open source is free, right?

00:14:02

It's free meaning that there's no initial monetary cost and it's free as in freedom. You can use distribute, um, modify the source code without going to ask the author permission. And something else you may have actually thought heard is that open source is free in puppy. Meaning, who doesn't love puppies? I mean, well, let's see, I'm looking the wrong way. But like, you know, they're cute, they make great friends, you know, they just make you smile. But having that cute companion that makes you smile can actually come at a high cost, meaning that they require a lot of work. You have to feed them, you have to clean up after them. And, but I mean, many people you'll see still have dogs despite the obligations, because there are ways to make it work. There's ways to make the process manageable. And open source is the same way.

00:14:49

Understanding those risks, those obligations, this is all important to ensure that you're able to be prepared, that the right people are involved, that the right tech is used, the right processes are being in place. So let's go ahead. I'm the lawyer. Let's go ahead and talk about the risks. Um, the main thing I want you to talk, take away from my part of my presentation today is just the fact that there are real risks using open source and someone in a legal or risk function that's not as familiar even in the business and technology function that's not familiar, you know, may have some hesitancy and, and it's easiest always just to say no versus taking the time to understand. Um, but hopefully after today's pre uh, presentation, all groups are ready to have that conversation because they're equipped with the right knowledge. So these next few, um, slides that we'll go through, have a little bit more information on it, and I'm not gonna cover everything.

00:15:41

The idea being that you'll get these slides afterwards, and you can look at 'em, is a reference deck. But what I'll talk about, um, here is there are two buckets of risks we'll talk about today. First is security and then legal. So jumping into the security, the, the takeaways there on the right are one, your developers are the first line of defense. You know, they're the ones bringing in the software. So they, they should be the ones managing it. There needs to be that you build it, you own it, uh, a mentality embedded into your culture. And this is actually consistent with the DevOps mindset. So also, um, unlike commercial applications that push those updates for you, your developers are actually responsible for pulling those, pulling those updates. And then since vulnerabilities can be found anytime, and we do see breaches and continue to increase that continuous detection and remediation is necessary, you do not wanna be part of that team that has to answer to your CEO and the executive team because there's been a breach, because you've waited over 1500 days to, um, resolve one's security vulnerability when really remediation and security role just means to upgrade, usually to the most recent version.

00:16:51

Now, they're also real legal risk. And, um, some things to keep in mind is you need a license. If you're using open source, that is your legal permission, that is the rules. And authors of software actually have over 2000 different licenses from which to choose. And then when we think about open source licenses, think about it in two general buckets. We have permissive and copy left. When it comes to permissive, generally the obligations are you wanna give attribution to that author, right? You wanna give credit. The other end of the spectrum is a little bit scarier for organizations, and that's the copy left licenses that in certain, uh, situations you don't even, you not only have to distribute and, uh, the source code and share the source code of the copy left license software that you're using, but you also have to share the source code of any code that it touches.

00:17:36

It's a very use case specific analysis. And then I also wanna point out that you're using that open source that you're bringing in as is meaning that you have no recourse against the author from what you got it. And that's also a tricky area to, to explain to your legal and compliance functions, because with commercial, you do have those, those reps and warranties. So, and because when we talk about remediation, we talked about security, right? You upgrade the version, it's a, it's a, it's a pretty straightforward approach with licenses, generally, licenses aren't gonna change throughout the, the course of the project. So remediation in that sense is a little bit different. It's usually replacing it with an open source alternative if you can't use it, something you write yourself a commercial alternative. Um, and then also remediation because the obligations are usually conditional. Meaning if I do this, if I use the open source this way, then this, right?

00:18:28

So because it's conditional, you can work with, um, your counsel to get, um, approval if the use case actually mitigates the risk that those obligations will, will take place. Uh, let's see. And sometimes, you know, if you don't know what the license is, instead of requiring removal, it's really easy just to get on GitHub to request confirmation from the author. That's all that you need. Also, if you wanna see if it's a license that you can't, you can't use internally for whatever reason based on your organization's, um, risk level, then you can just get online and get on GitHub and ask them, you know, Hey, let's, let's maybe add a permissive license to that. And then last but not least, we're all busy. So sometimes you just have to, you know, submit a pull request with that license that you'd like them to adopt. So it's just really quickly, uh, really easy for them to add.

00:19:17

Can I ask you a quick thing? Is that your picture on there?

00:19:20

I'm sorry,

00:19:21

Is that your picture there? Oh,

00:19:22

Yes, that is my picture. <laugh>,

00:19:24

You are a lawyer, right?

00:19:26

Last time I checked <laugh>.

00:19:29

And you work at a bank?

00:19:32

I do.

00:19:33

And you have GitHub account?

00:19:34

I do. I have my very, very own GitHub

00:19:36

Account. And you asked you to pull request.

00:19:37

I sure did.

00:19:38

You know, I, I don't care what people say about him, you are my hero.

00:19:42

Ah,

00:19:44

Topo iss too kind on stage. Thank you, Topo. But yeah, I mean, you know, I, I've actually generally had a really positive response when reaching out to developers. Um, it's, it's been, um, a really good conversation. Now, I, I may not always disclose or ever disclose that I'm an attorney, but that's only because <laugh>, that's only because I don't wanna stop the conversation before it starts, right? So this is just evidence that good conversations can happen. So speaking of the attorney, you know, you're what's the worst case scenario? And, um, we understand the high level security legal risks, but so what, right? Um, and these are just some general categories to think about. I'm not gonna spend too much time. If you want some more details about it, definitely come find me after. But these are the kind of things you wanna tell your leadership so that they have an idea of why it's actually an urgent need to actually get this stuff in place.

00:20:34

The first thing is trade secret disclosure. Your code is your secret sauce. If you have to open source your competitive advantage, you know what's left for your company's worth. Devaluation of prep patent portfolio, actually some, this is for companies that actually have patent portfolios. And so some, uh, open source licenses will have patent grants, meaning that you're actually giving out a free license, um, in certain situations for the functionality that covers the same functionality, uh, that your open source project does. And, you know, from a policy perspective, this makes perfect sense, right? So if I'm throwing out some software, open source software for the community to use, and then everyone's adopting and it's great, and then I actually have a few patents in my back pocket and they go sue everyone for patent infringement, we wanna avoid that. And that's what the licenses do. But you should be aware if you do have a patent portfolio, because the value of patents relate to your ability to exclude others from using it.

00:21:27

So it will devalue the patent portfolio, uh, mergers and acquisitions for the companies here that are looking to buy other companies or the companies here that are looking to hopefully get big enough one day to sell. Um, it's very important to be in compliance with your open source obligations. Um, open source compliance issues can delay a deal. They can devalue your company and they can kill a deal altogether. Security and, uh, security threats and potential fines that have heartbreak and FX right there, you don't want your your company to be a part of, uh, a national breach and get that much media attention and reputation risk, right? Your customers, your current employees, your potential hires, don't wanna work for, uh, a company that doesn't have, um, that doesn't have a good reputation for risk management, especially when you're an industry like ours with the financial services institutions, um, IP infringement, and now I know there's some network people here.

00:22:21

IP is actually intellectual property, not internet protocol, but, um, what, uh, typically litigation in the open source space, uh, has been from foundations, open source foundations, and they're really motivated by just making sure that people are compliant with their licenses, right? But we have seen an increase in the other, uh, category of people that bring litigation, and those are, uh, motivated by business and, um, more business related financial motivations. So this here is actually an order from a recent court case, uh, that recognized that the copy left license is both a license and a contract. And what this, uh, is important to keep in mind is that this opens up additional remedies that a plaintiff can bring against the defendant that they're suing. So something to think about here, what happens if because of a pending litigation, you know, you cannot use your mobile application, you cannot use your, uh, browser extension, and that's your main offering.

00:23:18

Or what happens if you're forced to comply with the, uh, the copy left licenses versus substituting it out and you have to open source, not only the open source you're using, but the proprietary code. That kind of goes back to the trade secret stuff we talked about. Both of these scenarios can have a big impact, so we understand who's using it, basically everyone here, and we understand some of the high level risk. So how do we think about, uh, tackling those risks? And I'm just gonna quickly put these all up just to show you that this is a framework to think about as you kind of start the journey, depending where you are, um, just making sure you know what you're using, assessing it, um, there's some solutions there that, that, that you can map, uh, different security vulnerabilities, for example, too. There are resources out there, and again, I'm happy to talk to you about this in more detail if needed.

00:24:05

But just think about is identify, analyze, and remediate, and then there's always gonna be that continuous monitoring and auditing, which can fit very nicely into your DevOps pipeline. And then mitigate with the governance strategy. Capital One has actually implemented, um, every one of these. So again, happy to to connect afterwards to talk about our specific journey with that. Um, I spent a lot of my time on the last one, education and training. And then I also included two links at the bottom that are some helpful resources that, um, are helpful to reference no matter where you are on your open source journey. So in addition to these resources, topos gonna actually help us understand how DevOps can help,

00:24:43

Right? So, uh, as open source helps DevOps, DevOps helps open source too. Uh, the first reason is you can automate. Now, do not automate a bad process, because if you automate a bad process, then you get bad automated process. So instead that, instead of that, look at your process and try to, uh, determine which one is more important to you or which one makes sense, and keep those and automate those. Shift left, uh, right, uh, from, uh, beginning of your pipeline, start looking for open source license violations and legal violations and mitigate accordingly. Frequent releases. If we need to patch something, we can always frequently release that patch as soon as possible. Instead of waiting for six months, 1500 days, uh, smaller batch size, that is reducing the risk. It can only release that particular library, uh, upgrade version to your production and collaborate and transparency is, uh, critical thing.

00:25:36

I, I mean, without that, you cannot, uh, go further and you cannot do it alone. You need to bring in engineers, legal, security, risk officers, or anyone else you may need along your way. Without the collaboration, there's no way you can get there. And from here, I say this, just using open source is not enough. And the reason I say that is because you remember I talked about our CI journey. Uh, we put all the libraries into our AVEN repositories, and while doing that, we found an, an, an incredible thing. There are libraries that look like open source, but they're not really open source in the sense that we modified them. Meaning that on that form, when he said, I'm not going to modify, actually lied, uh,

00:26:19

<laugh>,

00:26:20

Now I'm in trouble. Oh, sorry, <laugh>. Uh, so, and those modified libraries are in those, uh, uh, source code repositories. So what do we do with those? We cannot contribute it back. Uh, but we cannot use it because now we are stuck in time. So there has to be a way. So I started proposing this idea that we should contribute to open source, contribute back to open source, and that was pretty disruptive to make it more, uh, manageable. Uh, I, I thought that disruption is bad. So what did I do? I posted in our, in, you know, internal, uh, we call it pulse side, it's the social network within Capital One. An article block post should, should Capital One contribute to open source community? And the response was huge in the sense that it is against our policy. Well, let's change the policy. Why should we give her IP away?

00:27:09

We don't want to do that because Capital One pays my bills, right? I don't want to give my IP away and, you know, uh, do damage to myself. What are the legal implications? Implications? Don't know. Let's ask Jamie, uh, who will use your software? How does it matter? Maybe no one, maybe not even us, you know, but who knows? Uh, are you going to write another payment framework in open source? Maybe, I don't know. It depends upon the developers. What if we get the bad reputation? I'm like, wait, wait for a minute. We write code to handle people's money and we cannot write a code to log something in the log file. I mean, gimme a break. So we had a hot house from that, that at that point, the whole idea of hot house, that everybody comes together, all the decision makers, makers come together and try to resolve this issue, that, that developers are not allowed to contribute to open source in that hot house.

00:28:03

We basically decided that yes, we are going to make contribution. And that was a good day for Capital One engineers. Since then, four years, uh, thank you. We have contributed to open source 90 developers, 1 93 projects. Most notable are Angular, Ansible, Hadoop, block for JS Park, and so on and so forth. And we didn't stop there. We have our own open source projects also. Uh, we have been, uh, open sourcing our own internal projects for three years now. Total 31 projects most popular are Hygeia and Cloud custodian. Many of, uh, you, uh, already use these tool tools and contribute back, and we are not stopping there. Uh, we are actually, uh, so this is our open source page. Uh, there's another thing that we are doing. We are actually working with many enterprises to, what do you call, enter source, the next version of open Source. What that means is basically enterprises getting together and solving their own problems without having to rely upon commercial vendors. And we are calling that inter source. Uh, we have partnered with, uh, companies like Verizon, Walmart, America, American Airlines, and forming, uh, community around diverse practices.

00:29:12

So for us, being open source first results in a lot of benefits, right? A lot of benefits from Capital One, for Capital One, and some here are listed here. Um, but the one I wanna call out for you first, uh, and right now is, is the one we have up there in first, the culture of co uh, collaboration. Perhaps the most valuable observation I've made while being at Capital One and also working with Topo and the rest of 'em in the open source space has how much culture can influence behavior to do the right thing, the right thing for you, the right thing for your company, the right thing for your customers. And it's not an easy journey. Um, it, it's, there's a lot of questions. There's multiple risks to manage, there's a lot of lessons learned, which we're all kind of sharing today. But with the right knowledge, the right team, the right technology, the right processes, it's been an incredibly valuable journey so far. And the value is recognized, you know, beyond our development community. I mean, you hear me standing up here, talk about how valuable it is and even our business partners understand this as well. So we have a quote here from John Schmit, he's a product manager and our card line of business, and that just drives that home. And he's actually spoken at this conference, I think in London earlier this year. So, so it's really, really is a great community that we've established.

00:30:27

But no matter what you do, you will have an approval process. But, uh, here's my appeal. Make it easy, streamline, automate as much as possible. Not the bad processes, but the good processes. Collaborate often, and create a good engineering experience. And I'll, uh, end with a few of my favorite hashtags because hashtags are important. And I call these hashtag DevOps hashtags. Uh, the first one is Y-B-O-Y-O. You build it, you own it. Very important for your DevOps success. YBYS, you build it, you secure it. Thanks John. MVC, I learned it yesterday. Minimum viable compliance. Uh, that was awesome. GTGF, any idea what that is? Get together, go faster, which is this conference. And the last one is my favorite. JFDI. And uh, I'm gonna say this in front of you,

00:31:28

I'm not offended.

00:31:29

It means just do it. And, and, and the reason is thank you. And the reason is my final goal is to get the T-shirt. All of NUS change controls are full cycle and they're always approved. And I still don't have my T-shirt. I need your help. So if you have any ideas as to how to better manage open source contribution and consumption your enterprise, please let me know, and Jamie, and I'll be very happy to share our experience. Thank you very much. Thank you.