00:00:12

Hello everyone. Thank you for joining the session with us today. My name is Roger granddaughter and I'm a distinguished engineer at capital one and I'm Arthur Molson. I'm also a distinguished engineer here at capital one. We're super excited today to tell you a story about an internally built CICB pipeline framework grew to become a major inner source product at capital one. We'll share with you the lessons we learned that shaped our journey, and it can help your organization harness the power of inner source inside your company's walls. We'll start at the beginning. Many years ago, capital one followed a waterfall approach to software delivery. We had a lot of monolithic applications and they all ran in our own data centers. It was a painful process to really software into production and the deployment steps themselves were very male and error prone. It literally took months to get a release out the door.

00:01:09

If we fast forward to today, we operate a lot differently. We have agile teams following continuous delivery practices. We're all in the cloud, which enables our business to be more nimble and more importantly, we've automated our deployment processes so that they are safer, predictable and more frequent. The how exactly that we get to where we are today. Our digital transformation didn't happen overnight. It took a lot of hard work and effort from everyone. The CIC pipeline framework we built was an important part of that journey. If we hop into a time machine and go back to 2015, capital one had publicly announced at Amazon's reinvent developer conference that we were moving to the public cloud. And while our digital transformation was already underway at that time, the public cloud announcement was a big moment for our company and software engineers, including myself. We're super excited about where we were headed around that same timeframe in 2015 across the great lakes and Canada and our relatively small office near downtown Toronto capital.

00:02:19

One was standing up a brand new technology organization or what we call a software studio. And the studio was going to transform the way the capital one Canada business operates. The new studio and Canada had a startup feel to it. And we were going to do all the transformational goals. Our tech leaders had inspired us to do. We were going to automate all of the things. And so on a more personal note, I saw the new studio as an exciting new opportunity and challenge. So I packed my bags and I moved from Virginia to Toronto to join the team and be a part of the exciting journey. But as got started on our first few business initiatives and Canada, our developers quickly ran into a few roadblocks. As we worked, we had hired a lot of new developers. They literally needed to learn all of the things they needed to learn how to bill and migrate applications into the public cloud for bank.

00:03:16

They needed to learn DevOps principles and practices. They needed to learn how to run containerized workload. They needed to learn how to use open source software safely and securely, and how to automate manual processes. Each one of these on their own, as a challenge, but tackling them all at once. It's very hard for the average developer. And oh, did I mention that we're also a heavily regulated bank across three countries? That means there are many compliance requirements we need to go through for everything that we do. We soon realized that getting from point a to B, which in our case, was getting an application deploy and to production with automation was going to be filled with twists and turns. And our developer productivity was suffer unless we develop a solution that made their jobs easier. So with that startup mentality, I just mentioned, we decided to build a CICB pipeline framework that worked for the Canada business.

00:04:15

And we use open source software. We thought about the entire developer experience and how we could re-imagine re-imagine our software release process to make developers lives easier. We also made all of our new framework source code available inside of our company's walls, so that other developers in the office cause see the code and can, and could contribute features as well. This was an exciting and fun time. I recall a moment where a software engineer who I won't name, but they happen to be a Coasty care of this talk. They committed a bunch of new code changes to add a brand new feature to the framework. And immediately after the code change, we had developers running over to our desk, complaining that their bill pipelines were failing. Now it turned out to be a simple syntax error and we're able to quickly fix it. But this was an example of how quickly we were developing the framework and working directly with developers to gather their requirements and pain points.

00:05:15

Developers were our customers. It wasn't long after we built our first MVP that our Canada business launched its first microservice API on a public cloud with a fully automated pipeline. This was a huge milestone and accomplishment for a team. We knew it was a great start for the business with many more automated deployments to come and to give our framework our brand. We gave the framework and name and for this talk, we're going to call the framework. Bingo. We designed a cute little mascot for bingo because what developer doesn't love mascots stickers and free swag. The news about bingo quickly began to spread other parts of the company. Developer started talking to one another about how they enjoy the developer experience and how it solved the problem that they were facing. We knew where it was spreading because we had a dedicated slack channel for bingo.

00:06:11

And a key indicator we use for adoption was how many people had joined a channel. We will celebrate every milestone from 25 new members to 50 new members to a hundred, to 150. We were just seeing natural and organic growth, which was exciting, but we didn't know what was coming next. So as the framework continues to grow and take on new new users and new use cases, I decided to pack my bags and actually moved back to the us. But throughout this transition, we started to see a huge growth and bingo. We saw hundreds of developers starting to adopt and use the framework. Bingo was reducing the amount of time. It took teams to release software. It automated a way a lot of developer toil reducing the amount of time it took to deploy new software from months down to weeks, if not days. But we soon noticed that there were other similar pipeline frameworks being built across the company.

00:07:16

At first, I was surprised and be Willard at how many frameworks that have been built. What felt like duplicate effort turned out to be developers trying to solve a common pain point and to put this into perspective like any large enterprise capital, one comprises of multiple lines of businesses and given the size and scale of the company is easy for duplicate efforts to occur, even within the same line of business. So case and point, the Canada organization falls under the credit card division and we were seeing multiple pipeline frameworks sprouting up and just the car division. So within card, we began to naturally ask ourselves, why do we have so many duplicative pipeline frameworks? Can we consolidate to a single framework? Can we pull all of our talent together and solve the problem once and move on to more challenging problems? So this led to a healthy conversation between the various framework owners within the credit card division.

00:08:16

And we came together and a community event, we call it a barn raising. And as a community, we work together to figure out which framework we will standarize on for the credit card division and after much deliberation, the community decided to stand our eyes on the bingo pipeline framework that had been built in Canada. A key decision from that barn raising event was that we're going to turn bingo into a formal inner source product. And for those who may not be familiar with inner sourcing, the main idea here is that you take many of the concepts of building open source software and apply them inside your company's walls. So think about public GitHub issues, poor requests, project maintainers, along with some other processes and governance, but bringing them inside your company and an opera, optimizing them for how your company works. And there are a lot of great benefits to inner sourcing and that can unlock collaboration and productivity inside your company.

00:09:18

So benefits include code, reuse, knowledge sharing to help knock down those silos between teams that can often occur in a large enterprise promote higher code quality because more developers are looking at the software being used inside your company and overall just better innovation because everyone is focused on the same goal and feel a part of the solution. If you're new to inner sourcing, I would definitely recommend checking out the Dio rally book on how to get started with inner sourcing. Now, returning to our story, what happened after the barn raising events? Well, you can probably imagine the excitement of how bingo was going to become the standard for the card division, but also there was a lot of panic in our faces after we learned that bingo was going to be leveraged by the entire credit card business. A framework that had initially been built for a few hundred developers and a small office in Toronto was now going to be used by several thousand engineers and hundreds of application and Canada, us and the UK.

00:10:26

This was definitely an exciting time, but also there was this a lot of pressure to make this work. And at this point we had to Fasten our seatbelts and scale up our operation very much like a startup. The head just hit the jackpot. We quickly realized that we needed to replicate the bingo team and spread knowledge on how our framework works. So among the first things we did was we conducted large-scale bootcamp and in-person training sessions, and this enabled a couple of things to help educate more developers on how to use bingo for their business application. But more importantly, it taught those who wanted to contribute to bingo, how bingo works under the hood. So this education help teams adhere to many of the core principles that went into the initial development and success of the framework. We also ramped up our mob programming sessions, where we have multiple developers sitting physically next to each other, or even virtually between the us and Toronto developing new features and capabilities and real time.

00:11:33

We also found that there were other areas where we needed to make significant investments, including improving our documentation and expanding our office hours. Things were just moving very fast. And then we got hit with yet another curve ball, which brings me to act three when a process or product has been found to work well as normal and our large enterprise to figure out how to scale it up, especially the good parts. So remember when I said that capital one has multiple lines of businesses. Well, just like within a division, there were multiple frameworks, pipeline, frameworks across divisions. We had even more pipeline frameworks. And we, the question was asked again, why do we have duplicative frameworks across the company? What if we had just a single pipeline framework for the entire enterprise, imagine the reduction of duplicative efforts and the time saved by engineers. If the divisions is collaborated and work together, just imagine that well, that's exactly what happened.

00:12:37

All of the divisions got together to align on a single framework for the enterprise and collectively the organization does decide to take the framework that was built in the credit card division, bingo, and bring it to the enterprise as the base for all other divisions to build upon. And one of the good parts about bingo was this inner source spirit. We needed to take many of the lessons that we learned from inner source and card and quickly apply them for multiple divisions. We thought scaling bingo and just one line of business was difficult, but boy, were we wrong while bringing the divisions together? I remember a lot of conversations that went, something like this, Hmm, our divisional pipeline doesn't solve the problem that way we solve it this other way. And so working through all those differences and aligning on a path forward was probably one of the most challenging parts of bringing the divisions together.

00:13:38

But the more uplifting moments or those aha moments where, oh, you do it that way. We do it that way too. We should work together to remove the duplication. And we had a team in retail bank that needed a new serverless capability into the pipeline framework and that capability didn't exist. And their response was we would love to contribute that feature. How do we get started? So those types of open conversations is really what makes InnerSource super valuable and a large organization, and to help foster more of those collaborative conversations, we established what we call a pipeline council. And the council is as a group of leaders from each division to represent their divisions business needs to influence the direction and design of the share enterprise framework going forward. It was a great forum for everyone to align on priorities, use cases, roadmaps architecture, patterns, et cetera.

00:14:42

This really enhances the collaboration between divisions and like any popular open source product, the number of GitHub issues and pull requests against a successful InnerSource product will fill endless and become very time consuming. Additional contributors are definitely needed to really scale up. We had a community contributor who got to the point where they were consistently contributing multiple high quality, poor requests into the Bingle Rebos. They were also review other people's pull requests and provide feedback on their, on their poor costs as well. And to reward contributors who were actively engaged and consistently participating, we elevated them to become a trusted contributor. A trusted contributor is someone who has achieved a special status and our repos, where they are trusted to review and approve pull requests from other contributors. So having a path for users to become a trusted contributor has really enabled us to bring others in the community and give them a sense of part ownership in the product.

00:15:52

There's so much more to our story, but where are we today? Wow. I'm not allowed to reveal exact numbers. I can say that today. We have thousands of developers in our slack channel, which is a far cry from the small handful of engineers. We had. Initially we have hundreds of contributors and contributions coming in the framework across the entire company and the framework powers, thousands of automated pipeline builds every single day. Bingo has come a long way and it's amazing to see how it's grown, but what lessons can you take away on perhaps your journey to growing an inner source product?

00:16:36

Thanks for Roderick. Now everyone knows my embarrassing Syntech Sarah's story, but as Roderick was saying, our inner source journey was a winding and sometimes fraught journey, but we wanted to share it with all of you, because we'd love to inspire all of you to take on your own inner source journey and to help you along the way we would love to share and summarize some of the key focus points we think help make our inner source of journey successful and highlight some of the pitfalls we fell into. So you can avoid them yourselves. We feel that what we want to share is valuable, regardless of whether you're just starting out in your inner source journey, you've started thinking traction, attention and users or your product has gone enterprise wide, regardless of what phase you're in. We believe there's three key focus points for a successful inner source journey.

00:17:35

The first one is to look at your inner source project as actually a product, something that has continual upkeep, a roadmap and many of the product principles that many organizations understand and practice to use those on the inner source product as well. And one of the key aspects of a successful product is user focus. The difference with an inner source product is that the user is a software engineer. So developer experience becomes key to the success of an inner source product. And finally, as Roderick said, inner source was all about bringing the concepts of open source to an organization. One of the most important concepts of an open source product or project is focusing on the contributor and making their experience really great. So let's dive into one of these key focus points product from a product perspective in the early days. What you really want to look for is ideas, concepts, MVPs that scratch your own nature, scratch somebody, Sage, if a specific team or part of an organization, find something to be painful manual, not really a great experience.

00:19:00

It's very likely that many other people across your organization feel the same way as Roger described our journey to cloud. And that automation was something that we built to scratch our own itch via bingo, but ultimately scratch the itch of many other parts of the organization. As your product starts to gain traction and attention, you really wants to focus on setting down some core principles that serve as the pillars or foundations for your inner source product, as it continues to grow. In our case for bingo, we really cared about developer experience. So we would write down specific parts of that experience as core principles, for example, make sure that air messages are, are well-written and guide the user on how to fix them. And as you start to get to an enterprise scale, it's often possible, but changes can happen to the inner source product without a good discussion about the long-term implications, what the downsides might be.

00:20:14

And the overall impact of said changes. This is where we took learnings from the open source community and brought in the concept of RFCs requests for comments, many large open source projects like rustling, make good use of this, and we've tailored it for our needs. And as you start to also get to an enterprise scale, not losing sight of the details is one key focus point. That's also important because as you get to that scale, it's easy to lose sight of the individual in the medic metaphorical Hills, but focusing on that individual and making sure that their experience is still as great as it was in the early days, is what's going to keep your inner source product healthy and usable and sustainable into the longterm. So just to quickly summarize from a product perspective, as a key focus point for say, a successful inner source product, look for products that scratch your own niche.

00:21:22

As you start to scale, create some core principles. And as you get to enterprise scale, consider introducing RFCs and making sure you don't lose sight of the details. Let's move on to users. As I said, one thing that's unique with inner source products is that they're focused on the software developers. So early on what we found was key to our success was making sure we provide that white club support. This helps build that foundation of advocates who not only had a great experience because we sat with them and troubleshooted with them. We also gave them some insight into how the product works so that they become better advocates in the future. As you start to scale, what we found is the manual touch points, the places in our product that required manual intervention start to really grind on your ability to scale. One story that comes to mind is actually our onboarding process required 20 to 30 manual steps that engaged both the developer, trying to use bingo and us as the core team.

00:22:38

So we took the time to automate that entire onboarding process. And it became a slack app that took very little time for the developer. They had a great developer experience and took no effort on our end. So we were able to focus on new features and bug fixes. And as you start to scale to the enterprise level, when I mentioned not losing sight of the details, one of the big details is actually user happiness. And the way to gauge that is to use an industry practice known as that promoter score NPS, this will start to help you see how do others perceive your product and can give you the ability to narrow in and focus on specific individuals or groups and have focused empathy interviews, or other mechanisms to get more details on why they feel a certain way. So to summarize, focusing on the user is critical for an InnerSource project success.

00:23:43

What we found is that, uh, providing white glove support in the beginning is a really great way to build up advocates. As you start to grow, automating a way manual touch points so that the developer experience is maintained and is even better. And you free yourself up. And as you get into enterprise scale running quarterly or twice a year, NPS surveys moving onto contributors. As I said, contributors are the lifeblood of an inner source product. And the way to engage contributors early on in your product is by building it in the open, being transparent, doing everything in GitHub or your internal source repository, but making it public like Roderick was mentioning, uh, having those discussions and code reviews public as well. I can still remember early on, we would have random software engineers from Canada, join in on a code review and they would provide really valuable feedback.

00:24:47

So it's a really great way to engage contributors early on. As you start to scale, as Roderick mentioned, the in-person training and boot camps, we found those really valuable, not only giving them the technical details, which, you know, we can kind of do that asynchronously, but instilling the principles and the culture of the inner source product. That was the most value that we saw from the in-person training and boot camps. And as you start to get to an enterprise scale, providing a trusted contributor model, a path to move up the ladder and take on full ownership of some parts of your inner source product or ecosystem, this lets you as the core owner move on, but still feel comfortable that it's left in good hands. And as you get to enterprise scale recognition, which is important in all stages becomes even more important and recognition doesn't have to be complex, sending an email with a thank you to the contributor, but CCing, their manager goes a long way.

00:25:56

You know, it's not open source, we all have bosses. And when our bosses know we're doing a great job, you feel great as the employee and you want to do it again. So if focusing on contributors as a key focus point early on developing the open as you scale using in-person training mobbing sessions, bootcamps to onboard new teams and at the enterprise scale, introducing a trusted contributor model and not forgetting a book recognition. So those are the three key focus points that we think help make a successful inner source journey. But of course, along the way, wait, bump your head. And if Roger can, I could jump into that DeLorean at the beginning and travel back in time, four years, some of what we would tell ourselves, whether we'd listened to our future selves or not, I don't know, but when we would warn ourselves about remembering and focusing on and making documentation a first-class principle in your inner source product, make it your definition of done.

00:27:03

Just make sure that documentation is not only written, but kept up to date on a regular basis because this comes back and haunts you in the future. As we grew to a larger scale, started to gain traction. We'd remind ourselves to always keep up with the latest version of whatever tool we're using, not to put off upgrades and to keep track of the code quality and not forgets that the health and quality of the underlying code is really what forms along with the principles, the foundation of your inner source product. And as we started to get to enterprise scale, we would remind ourselves to be open to different contribution models that maybe what works for one line of business doesn't work for another and just be more open and empathetic there. So these are some of the kind of pain points and pitfalls that we ran into.

00:28:04

So more documentation at the very beginning and really throughout reminding ourselves to keep up with code quality, keep up with versions of the tools that we're using and being open and willing to experiment with different contribution models. And of course this wouldn't be an enterprise conference. If there wasn't something that looks like an Excel spreadsheet. So here's a quick summary of everything we just talked about. Thank you again for listening. Uh, we really enjoyed having you here. I'm Arthur Molson. Feel free to reach out to me on Twitter or LinkedIn, if you have any questions. Okay.

00:28:45

And I'm Roderick Randolph. You can find me on LinkedIn, if you have any additional questions, thank you again for listening to our town.

00:28:54

And here are some additional resources. Of course, the slides themselves. You can find there, but also linking to a lot of really great knowledge. That's been built up around how to run and introduce inner source to your organization. Thank you again.