Driving a Tech-led Reimagination of eBay Through DevOps (US 2021)

One of the original hyperscalers, eBay helped pioneer modern operational techniques like sharding, circuit breakers, feature flags, and distributed tracing, and in 2020, eBay’s data centers turned in record cost effectiveness and availability. Also in 2020, however, the company needed to confront waterfall technical practices, years of legacy software and technical debt, slow product delivery, and a challenging competitive environment. This session outlines eBay’s technology-led reimagination begun in 2020, specifically the cross-organizational Velocity initiative to improve eBay’s ability to deliver value to customers. We started by characterizing the breadth of the problem, which ultimately spans culture, organization, people, and technology, as well as every stage in the product development lifecycle. Using value stream mapping to identify constraints to flow, we decided to focus our initial efforts on improving software delivery across the board, because we recognized that eBay’s ability to deliver software rapidly, safely, and repeatably was a prerequisite for every other improvement. In addition, since one particular area of the site was a bottleneck for numerous business initiatives, we also focused on modularizing and modernizing its architecture. Because our focus was on software delivery, we adopted the Accelerate metrics to measure success and look for opportunities. We launched ~10 independent tracks -- from build time to continuous integration to automated deployment -- and hand-selected pilot teams to work with in tight feedback loops. We have also embedded subject matter experts directly in product teams. We further use team-of-teams weekly meetings to share learnings and reinforce continuous improvement. Explicitly to break down silos, the presenters are co-leaders of the initiative -- one from the product engineering side and the other from the tools and infrastructure side -- with entirely shared goals and metrics. The initial results of this initiative are already bearing fruit, and we identify -- and bank -- new wins every week. With support from the top, and excitement from the grassroots, the Velocity initiative has galvanized teams to question the status quo and look for opportunities for improvement. Equally importantly, working collaboratively and breaking down silos are paying second-order cultural dividends. We have a long way to go, but this session will provide actionable insights for other organizations going through similar journeys.

usplenarylas vegasvegas2021
RS

Randy Shoup

VP Engineering and Chief Architect, eBay

MW

Mark Weinberg

VP of Core Product Engineering, eBay

TRANSCRIPT

00:00:12

Welcome to the afternoon plenary sessions. So I have known and admired the work of Randy Shaw for a decade. I met him at jazz humbles flow con conference, and maybe the best piece of evidence of just how much I love his work is that Randy is one of the most cited people in the DevOps handbook. It included the work that he did as engineering director of app engine at Google and as chief engineer at eBay over a decade ago, I am so excited about this presentation after nearly a decade. Randy Sharp is again at eBay this time as VP of engineering and chief architect. And I've always loved how Randy thinks about solving problems to see this problem solving dynamic, put in service of increasing productivity at an engineering organization that has thousands of developers. It is truly are inspiring. I'm so delighted that he will be presenting with his colleague mark Weinberg, VP of core product engineering, who is a technology leader of the entire product engineering organization between the two of them, their areas cover almost everything that gets done at E-bay engineering. So here is mark and Randy.

00:01:21

Okay. Hi, I'm mark Weinberg from eBay. Um, welcome Brandy eight shop and I are here today to talk to you about how we're using dev ops at eBay to transform our engineering. Um, so first I thought I would just start off with the problem statement. What are we trying to solve? Well, frankly, uh, we are too slow as a company and we lag industry leaders in terms of our engineering velocity. And, and why does this matter? Well, engineering velocity leads to obviously better customer experiences, stronger business results, more engaged and happier employees, which we all want. And if you think about it, if you're a developer, you want to write code, you don't want to sit in a wait state. You want to just work on the code, work on the product. If you're a product owner, you want more features and value for your customers.

00:02:12

And if you're an executive at the company, you want to stay ahead and beat the competition. So the bottom line is we need to move faster. And so our mission is to turn this around and make engineering velocity, a competitive advantage for the company for us getting a little faster, just won't be good enough. There are competitive threats everywhere. We have threats from companies, big, bigger companies like Amazon and Walmart, but there's also new startups like Shopify and stock X and others. So bottom line is we just have to get faster. So how do we get here? Um, Randy and I spent the first three months at eBay, basically serving the state of engineering velocity. And we looked at the entire landscape. Uh, we talked with engineers and engineering leaders across company and a high level. What we found is that there are just basically systematic systemic challenges that have accumulated over many years of the company.

00:03:08

And this ranges from code and architecture. There's lots of tech debt and monolithic code, a missing tools and infrastructure we had, uh, had a low quality staging environment. Um, some poor processes, lots of PR submitted with too, too many changes, involved, long loop feature branches, slow code reviews. And, uh, you know, only yearly site-wide upgrades of frameworks and platform elements and lots and lots of major team dependencies that effectively created distributed monoliths in the, in the code. Um, so as a result of that, there's no silver bullet, um, fixing these issues is going to require improvement across many different areas and span many different disciplines in the company from engineering, finance, customer support, even at the executive level. Luckily, uh, these are well known problems with well-established solution, solution patterns. Uh, many companies share these challenges. Perhaps many of you at your companies have these same issues, uh, which is great. So we get to learn from each other and apply best practices and just effectively help each other get better. So today, Randy and I are here to share what we've done and what we've learned. So let's go onto the next slide.

00:04:27

So most of you probably know about E-bay E-bay is a global e-commerce leader. We connect millions of millions of buyers and sellers across the globe. Um, and in 2020, we are a $10 billion plus a year revenue company where there's 150, 59 million active buyers on the platform, 19 million active sellers. Um, so it's a very vibrant and high traffic high-scale site and application. We have roughly 12,000 employees across the entire company. So just introducing myself a little bit better. I work in the core product organization. I'm responsible for effectively leading the technology across many different parts of the product org organization. I also run a few teams. I've a team doing a big feature for eBay called stores. I run a small little mobile team and I run our, our planning team that drives our, our quarterly and yearly planning for the entire, uh, product organization. Um, and so let me just introduce Randy poop, give you a little bit more information about himself.

00:05:35

Yeah. Cool. So thanks mark. Uh, so I'm, Mark's partner in crime for all this stuff. So I'm the VP and chief architect for the company. Uh, and my areas are the sort of platform and developer experience, uh, architecture standards across the frameworks and mobile and all sorts of places at eBay. Um, I have a sort of stable of enabling architects that we deploy, uh, to individual, which we'll talk about later. And then I also own the cross-functional product management organization. So together, the two of us are able to like apply, apply our leadership in like reframing and reprioritizing, a bunch of priorities of the company. Uh, we're connecting teams, you know, from Mark's organization and mine to, uh, figure out a, to help have them help each other to unblock each other. We're sort of, we have the capability to like encourage teams and permit them to take more risks than they otherwise would.

00:06:25

And we have both the ability to sort of suggest to teams, but also to implement mandates when that's required. Um, so, uh, the, the next thing that we did was we did an assessment of, as mark mentioned of, you know, what the situation was at eBay. And so standard's kind of value stream mapping approach. So a couple of teams that are, that are a cross section of all across all of eBay. Um, and so looking at, you know, all of the phases that go from idea to customer value. So planning is like when we go from idea to, we start a project development, as we go from project to committed code delivery is committed code to deployment on the site. And then, you know, once we've deployed, we need to iterate to make sure that we continue to drive customer value. And we found issues at every stage.

00:07:11

So from the planning perspective, there's lots of coordination, lots of inter-team dependencies and almost every team at the company has too much work in progress. In terms of development, we found that developers were suffering from really slow build and test time. They were doing lots of context switching and being, uh, kind of, uh, blocked in wait states, queued up and wait states. We have 26 years of architecture behind us and three, uh, three generations of infrastructure. So, you know, lots of highly coupled architecture, uh, there, um, we don't yet have a tradition of sort of service contracts between the services, even though we have many, many services. And of course, lots of hidden work from a delivery perspective, you know, not every team has a good end-to-end automated, uh, deployment pipeline. Um, lots of issues that mark, that mark mentioned at least several years ago in the staging environment, lots of teams do manual testing without automated rollout.

00:08:03

We're now introducing Canary deployments and a better usage of feature flags, but there's lots of opportunities there as well. And then finally in that iteration phase, you know, we found that teams don't necessarily have an end to end monitoring of their systems, but also the kind of business metrics and customer experience, lots of issues in tracking what customer sees, and then what we would frame as a dysfunctional experimentation capability. Like we actually have a great experimentation platform. Some teams don't use it at all. Other teams, uh, overuse it. So lots of, uh, targets then to target rich environment. But what mark and I have decided to do is focus on, uh, the laser focus on the software delivery and a little bit into the software development area. Why, because this is the bottleneck. Um, the, you know, thinking of the theory of constraints, idea, um, software delivery is, is currently the bottleneck at eBay and our ability to make improvements in all these areas.

00:08:56

And if we can unblock and unlock software delivery at eBay, it makes everything else that we want to do fast, uh, possible by enabling faster change and reducing the cost of change. So if I can deploy multiple times a day, instead of a couple of times a month, now I can make architecture improvements more experimentation, more changes for customers. So now I'm going to talk about how we're measuring our success. And then after that, I'm going to hand it off to mark to talk about what we've actually done. So in terms of measurement, we're doing, we're trying to improve software delivery. So we're using the door or the accelerate metrics as my friends at eBay, see all the time like I'm showing the accelerate book almost constantly. Um, so here's how we measure ourselves the standard metrics. If you look at eBay overall. So before we, and if we, uh, started doing this philosophy initiative that we'll tell you about, we're basically a medium performer across, uh, across the board.

00:09:47

And just to, you know, give you a sense of where we've been able to go. And just this year in, in the velocity work is the pilot teams that we're working with has actually, we've actually been able to move all of those teams into the high performing area. So, uh, pretty exciting work. We've been able to triple our deployment frequency for those teams, multiply that lead time for, or reduce the lead time for change two and a half times, keep time to restore service, essentially the same, and we've actually improved the change failure rate three X. So, um, so pretty great results. And now I'm going to hand it off to mark to tell you how we do it.

00:10:21

Great. Thanks a lot, Randy. So let's, let's go ahead and go to the next slide. So how are we going to get faster? So what we decided to do was to focus our efforts on a select number of what we call pilot domains applications within those domains and platform tracks. And the reason for that is we didn't, we were learning, we didn't want to start with everything all at once. So roughly about 10% of eBay's active app applications are in this pilot. Um, and then for each of those, we really wanted to focus both on short term wins and longer-term capabilities cause they both matter, um, on the short-term side. For example, if we can say, save five minutes on a task for every engineer, every single day across the entire engineering organization, that turns into actually a big win. So we want to do lots of those things, but there are also larger changes, like a refactoring code rearchitecture these are big, have a bigger impact.

00:11:18

What of course carry more risk, but there's a few of those that we do need to kind of focus on. I'll talk a little bit more about that. Um, driving improvements. It, this is the core of kind of what we're, we're focused to, to Randy's comments, developer productivity, um, the, the sort of daily, uh, build debug tasks cycle that developers are in most days. So really improving our, our build times, our server start-up or PR validation times, that's a huge focus for us. Um, and then sort of in software delivery, um, improvements in software delivery are going to make everything faster. So we just have a ruthless focus on process automation. Uh, as a company we'd still did lots of manual testing, lots of manual steps. And so we're trying to automate things like our load and performance testing, our site speed tasks, and then we do something called partner sign-off.

00:12:10

So when you have one team dependent on another team, another team will run tasks. And a lot of that was sort of manual back and forth through emails. And so we're trying to automate those processes, things like security, localization, accessibility, testing, automating those things. So we're real focus on that. And then also focusing on faster deployment. So, uh, Canary deployments using techniques like traffic mirroring, uh, those really help us get, you know, get deployments out quicker. And then, uh, as I mentioned, things like, uh, instrumenting the code more focusing on alerting and proving our observability so that we can be confident to roll out faster, knowing we can roll back fast as well. So eBay now, like we're managing payments for customers. And so, you know, when you're talking about customers money, you have to be able to detect and resolve issues extremely quickly. So now, you know, now we're much better able to do that.

00:13:06

And then on the architecture side, we focused on areas that have high active development, uh, places there there's high customer usage. So the call out here is, uh, we have a page and a page on the site in our app called view item. It's the primary, uh, experience for the product, um, hundreds of millions of views a day on that page. So if we can make velocity improvements there and allows us to improve the product, you know, much more rapidly, and these, these areas also have lots of team dependency. So unblocking, those dependencies is a big focus for us. And there also the, the old code was very brittle and hard to test. So that's where we're putting our focus. So we can skip to the next, next slide, Randy. Um, the way we think about this as sort of in terms of more horizontal platform tracks, and we call out some of these here.

00:13:55

So these are tools and infrastructure to better support our builds, our CGI, our staging environment, even things like educating and training our engineers on techniques and patterns. And then we have a bunch of these pilot domains, which cover big areas of the product, things like selling search and ads, and these teams work very closely together, um, to come up with ideas of how to make improvements, implement those improvements, and then share across all the domains to make everybody faster. So let's go ahead and go to the next slide. Um, this has really truly been a collaborative effort across our technology platform teams and our application teams. These teams work extremely well together, very eternally. This didn't use to happen at the company. There was mostly kind of walls between the two. So we've really kind of done a good job to drive the collaboration across.

00:14:51

Um, as Randy mentioned earlier, we embed senior architects within these teams and they've really helped to coach advise, even write some critical code and that's made a big difference. Communication is key Randy and I meet every single day to sort of talk about issues, come up with solutions, work on our plans. And then we drive a weekly scrum of scrums meeting where individuals and teams learn from each other on things that have been working and maybe things that they could use help on Randy. And I do weekly, deep, deep dives with each of these teams to coach, to, to push them, to find gaps and come up with, uh, solutions to problems. And then we also regularly do a monthly operating review with our executive team to keep them up, up to date, to keep them engaged and interested and curious about what we're doing because we need their support. So, Randy, why don't you maybe talk about the other things that we have?

00:15:45

Yeah, sure. Thanks mark. Yeah. So from a measurement perspective, um, so we're using the four key metrics as the way to measure our progress. And so one of the things we did almost immediately was add to our existing development dashboard, the four key metrics for every app. It turns out we had all the basic data, but we hadn't put it together in, in that form a week. And then we've continued to iterate on that. So we're at a granular visibility to the entire delivery pipeline. So for every deployment, by any app we can look at, oh, okay, well, how long did it take to build, how long did it take to roll out to the first machine to the last machine? Uh, was it a rollback was, uh, was Canary deployment used? Uh, we keep adding more and more capabilities to this, and it's been super helpful for us to see the kind of overall, uh, visibility, but also super helpful for the individual teams to debug and optimize their processes.

00:16:33

And then also for each of those platform attracts that, um, that, uh, mark mentioned, we have sort of lower level input metrics, right? So here's our goal for improving P 95 build time. Here's our goal for improving P 95 startup time. Here's our goal for improving and to NPR validation time, et cetera. So we track ourselves at the individual level and at the sort of overall program level in terms of iteration, like this has been the huge, uh, unlock is that when we went to teams and we said, Hey, we see you're develop, you're deploying like, you know, once a month, twice a month, what if we asked you to deploy every day? They'd be like, oh my God. We say, well, tell us why you can't. And they give us a big long list of, you know, like 10 or 20 different things that they need to do that they can't do.

00:17:16

We're like, great. We're going to knock down the first three. And then we're going to knock down the second three. And oh, by the way, this other team has already solved, you know, seven, eight and nine. And so like having the conversation from the perspective of here's the goal we want to achieve and tell us what we can do to help you to remove those impediments has been the huge unlock. And that's been, that's been the driver for all this collaboration for each of those things that we're doing. We do a very tight, you know, Deming cycle of plan, do check act. So we try a thing with one of the teams, if it works great, we start rolling it out to everybody else, you know, in the velocity program, but also outside. Uh, if it doesn't work, then we stop it and try something else. And then as mark mentioned earlier, we do have a pretty big focus on education because there's a lot of opportunity to help educate the thousands of developers at eBay on these new, modern ways of doing things with, you know, domain-driven design and TDD and automated testing and all that.

00:18:11

So, um, so here's some quick results. So this is the big, uh, the big thing that, um, you know, really matters to the execs for the pilot teams that have been involved in this initiative. We have been able to double their productivity. So we've moved the metrics, uh, the adore metrics, but like it has shown actual results in terms of doubling the productivity of those teams. What does that mean? It means that holding the team size, constant, so same team, same members, the teams are delivering twice, two times or more the number of features, bug fixes, et cetera. And so, again, as mark mentioned, those pilot teams roughly represent 10% of the actively developed apps and services at eBay. And again, as I mentioned, we've improved for those teams, their deployment frequency by three X lead time by two and a half X and our change failure rate by three X, it might be mark can talk about what we did.

00:19:01

Yeah. That that's, it's, it's really been fun to kind of see the results. Um, the, the first thing is we've talked about is putting these metrics in place. And I couldn't recommend doing this more using industry, these industry standard metrics. It really kind of stops the debating that that happens of, you know, are these the right metrics to use. And are we sure that these tell us if it's working? Like we hear that a lot, and it's, it's just been great to be able to kind of point to those and research behind these and, and show the correlation of how improving these, these metrics that leads to better productivity. Uh, so I couldn't recommend that more. We, we focused on removing bottlenecks, as Randy mentioned, we just keep, keep tearing these down with the teams and we knocked one off and we ask them, okay, what's the next problem that you have?

00:19:45

And we keep it, keep doing that. And it's really made a big difference. Um, we spent a lot of time focusing on nuts and bolts stuff. So, so our build startup in PR validation times, just as a quick story. Um, when I started at eBay, I like as I'm a developer myself, um, I asked to do a build of one of our larger components and I, I sync the code and did a build, and I found it took an hour to do a build. And I was shocked by how long it took and started talking to people and, and through a lot of work in both sort of our tools and really kind of paying attention to these things and just doing the work that 60 minute build time. Now it takes four minutes. So, and we've done that a number of times, it's just kind of amazing at the improvements we've been able to make in these sort of core areas, PR validation, times another other case, we had a team that was taking an hour and 45 minutes to do, uh, you know, to validate a PR that's now down to 10 minutes.

00:20:42

And so you add these things up, it really makes a huge difference. And then as we've talked about just our staging, really getting that healthy, making sure that we have good quality data, that's privacy, clean, good components in that that are reliable. It can be trusted. And that, you know, just having the engineers treat our staging environment with care and like it's a production environment. So we can trust that when we run something in staging, we know it's a good representation of how the thing will work in production. So that's been a big difference. Um, automation automating our op our site-wide upgrades, automating our testing, automating our deployments. We still have work to do in this area. We've made just a ton of progress, but, you know, our aspiration is to get to one-click automated deployments for all of our active applications. We're not quite there yet, but we've put a big focus on that.

00:21:32

And then other things like just some more of our definition of done processes around, um, partner sign-offs, uh, doing proper code reviews, just paying attention to these things, not letting code review, sit around for a couple of days, you know, having just sort of a regular recurring rhythm on, you know, just jumping on code reviews, just things like that have made a big difference. And then lastly, um, what about a year ago we were doing our mobile releases just once a month and that, that just led to all kinds of problems. And now we're down to doing weekly mobile releases and that's just improved our quality people. Don't rush to get a change in because they're worried about the next train, not leaving for 30 days. Uh, they do things right. It also allows us to do faster, hot fixing of a problem. It results in smaller batch sizes.

00:22:22

And so that has been a huge improvement for us. So let's go to the next slide. Um, sort of a lot of this stuff has changed our culture. Um, you know, we look at metrics, we, we do this regularly. Um, teams are inspired to work with each other and help each other. Um, teams don't want to go back to the old way of working. They, they feel like it's working, we hear that a lot. Um, and we have actually teams that aren't in the pilot that they're asking us, can we, can we participate? So there's a lot of demand for, uh, some of the improvements that we're making. It was really kind of fun. And then on the collaboration side, just really creating a no-blame culture. And I think, um, that's really paid off and I think people it's made the environment or fun to work in. Uh, people find a problem. They want to work together and fix the problem. Um, we do a lot of partnering with teams where maybe we didn't have close collaboration before, like our security team, our Sox compliance, our accessibility and localization team. So, uh, you know, that's just more fun to work in that kind of environment.

00:23:27

Yeah. So from a, I so much fun, so I'm from the community and sharing perspective. Uh, this was something I was not expecting at all. So the standard eBay model is like my teams on the core technology side, produced tools and infrastructure, which are then consumed by the product engineering side that mark works. And what we found as part of that still is true. But what we've also found is that teams in the product organization are automating their own workflows. So they're writing little tools to help remind them to finish PRS and, um, uh, and code reviews and to do a bunch of the security, uh, to the, uh, performance testing, to do the accessibility testing that they need to do. So teams are automating their own stuff. We give them space. It's now multiple times a week to like do T demos of the new tools that they put into place, new practices, a big opportunity to share what they've done with other teams.

00:24:15

And there's a lot of team pride associated with that. And that sharing, um, is something that never happened laterally between those teams. So it's been a great, a great, uh, injection of community and collaboration into the organization and then last, but very much not least like we, we really benefited from a lot of executive support and engagement. So, um, just as I'm a returnee to eBay. So as our CEO, um, and, uh, he constantly highlights at the company all hands and at exec meetings and at these operating reviews, just how important this philosophy initiative is. And so he says many times, uh, this is the most important initiative at the company. And then the, you know, like, oh, for mark and Randy's like go faster. So, you know, it cuts both ways. Um, here are some of our current challenges. Um, so from a program outcome perspective, you know, again, we've been really good at and improving the team level metrics, but what we haven't yet been able to do, and we're hoping to do in the next phase of this work is to improve overall E-bay outcomes.

00:25:15

And we've got some ideas about how we can leverage Nickerson's flow framework to like, think about those outcomes and really measure them in a good way. Um, but that's something that's, uh, that's a remain a remaining thing to do our challenge for us as the initiative team is, you know, there's never enough time and resources to do what we want. So like we have a big long list of like all those impediments. We want to help teams on block. And we're just, under-resourced from the perspective of the platform side, but we're frankly also under-resourced on the product engineering side. So like, we need to make sure that the teams that are, that that are involved are like continue to, you know, retain their commitments to improving their velocity. In addition to all the other things are asked to do, uh, like, you know, security patches and oh, by the way, features and, you know, customer value.

00:25:56

Um, and the other thing that we found relatively recently, but was quite interesting is that we found that a lot of the teams, um, their representatives as part of this program, tend to come from the QE side of the organization rather than from development. And that's been great because those teams are about those people are about quality and they are about, uh, the pipelines. Um, but what we find when we find that we, uh, when we're trying to like move a little bit more upstream, more trunk based development, uh, change the practices of the development team themselves. We're finding that we need to, you know, bring in the engagement of the development leaders too. And that's again, a learning for us. And then the last thing, which is a little bit, you know, for mark and me, is like, we're pretty overtaxed. So at the moment, you know, mark and I are doing a lot of, you know, high level stuff, but a lot of, you know, deep dives with individual teams and we need to figure out how to, you know, scale ourselves and delegate a little bit more. So I'll hand off to mark to talk about other, other challenges here.

00:26:50

Yeah. W so it, we see a lot of people focusing too much on the metrics and, and losing sight of what we're actually trying to achieve. Um, there's, there's also a little bit of gaming of the system. You know, people try to make sure that they're hitting these metrics. Things like, you know, we, we want it, we want actually more pilot apps in the program. Uh, but when we add new apps, then maybe our deployment frequency numbers go down or our lead time for change goes up, uh, because they're starting. And so we hear that, that sort of a lot about the metrics. And then there's sort of this notion of, um, fear of failure and the consequences. Like if we go faster, are we going to create quality issues? And if we create quality issues, are we going to get in trouble? And so I, you know, obviously we're trying, we're trying to teach people that going faster doesn't necessarily mean that quality goes down.

00:27:44

In fact, we're seeing it actually go, it gets better. Um, but also just creating this environment that, that people understand that the fear, the, the going slower causes has consequences that are probably greater than us, you know, having a deployment, you know, that we had to roll back or something like that. So that fear of failure trying to get people over that is really important. And then we still have some people that don't necessarily believe in the approach. And so, you know, there's sort of engagement with those folks and how we're doing things, why we're doing things and teaching them why this works and how it has worked at other companies and how it's led to success. So, uh, that's just the, it's a big company. There's a lot of people. And so that education part is an important part of, uh, kind of the, the area that we, we need to get better at. So,

00:28:35

Okay. So I'm just going to close with what we hope the future is going to look like. So, you know, by contrast to the current state of our lifecycle, what we'd like our future goals to be. So here's, you know, what's our north star from a planning perspective. We'd like to be doing rolling planning with many small cheap experiments that we're doing all the time. And then when, if an only, if we find some value out of that experiment only then do we double down with a big, massive cross-functional project from a development perspective, we'd like teams to be doing small batch sizes, maybe single piece flow, you know, one single PR or one single commit all the way to production. We want fast build and test iteration. We want at least daily mergers and deploys, and we'd like a more decoupled architecture as we go forward from a delivery perspective.

00:29:19

Again, what we'd like as a fully automated test and deployment pipeline with, with one hour commit to deploy. And we'd like to be doing a lot more iteration and production using feature flags and last, but very much not least on the iteration side, you know, we'd like to do way more end to end monitoring. We'd like to have tracking everywhere, again, many small cheap experiments with rapid feedback on those results. So the help that we're looking for is we're going to be scaling this initiative from 10% of applications to hopefully 50% really soon. So we'd really love people's experience around taking an initiative like this and scaling it much more broadly. Again, as we mentioned in our, in our, uh, in our challenges, we we'd like to help on inspiring and motivating, uh, the engineering managers, like it's easy to motivate the developers they're self-motivated, but for all this work, it's actually easy to motivate the execs. It's the middle. That is a little bit of the challenge. And then also, you know, we have great executive commitment and sponsorship at the moment, but again, we'd love, we'd love people's thoughts on how one sustains that commitment to this kind of initiative over the longterm. So, as we mentioned, there's a huge amount of executive sponsorship for this work. And so we're lucky to have our CEO, Jamie to tell us about how this matters for eBay.

00:30:33

Hi, everyone. I'm honored to be here today among this incredible group of tech leaders. You know, when I came back to eBay in 2020, I shared my vision for a tech led reimagination for the company. One that would propel us forward and help us set the stage for the decade ahead. The cross-functional velocity initiative that we kicked off this year led by mark Weinberg and Randy Shap is a key part of our strategy, underscoring, our work to create environment where both engineers and innovation can thrive. We've already seen this initiative, double the productivity of the teams involved and directly improve our ability to deliver value to sellers and buyers around the world. It's truly incredible. You know, I think it's helpful to explain how this success has been achieved. First, we started with clear goals and metrics and a collaborative approach across both the product org engineering and technology platform organizations.

00:31:28

And throughout the process, the teams have zeroed in on the impediments and bottlenecks in our software development system, working to eliminate them one by one. We've also embedded strong architects into our teams to help improve the team's technology, their practices, and their ways of working overall. It's great to see how excited and inspired all the engineering teams are to be part of this initiative and how they're continually working to improve our velocity across the entire technology organization. I've asked mark and Randy to scale this initiative even further next year. And I can't wait to see what they and their teams are able to achieve. Thank you.

00:32:06

So thank you very much. This is a lot of fun for us, and I hope it was valuable for you all. Thank you.

00:32:15

Thank you, mark. And Randy, and my heartiest, congratulations to you and the entire E-bay engineering teams that led to that amazing message from E-bay CEO, Jamie . I think it is a phenomenal Testament of what you and the teams have done and shows just how important the work of this community is. And here's my request for all people interested in speaking at future devil's enterprise summits. If you are in a position where you can get your CEO or COO to share a similar message, I definitely want to hear from you, the team from nationwide building society was first to pull this off, followed by American airlines, Fannie Mae and eBay is now the fourth. And I am confident that they will not be the last.