From 6-Eye Principle to Release at Scale – adidas Digital Tech 2021
From 6-Eye Principle to Release at Scale – adidas Digital Tech 2021
VP, Digital Tech, adidas
Senior Director- Head of Digital SRE Operations, adidas
SRE Champion WnM Services, adidas
Thank you, Peter. Our second presentation today is from Adidas four years ago. The tea from Adidas attended this conference and they left inspired by the presentation from Jason Cox from Disney. And I have been so amazed by the audio DAS journey since then, Fernando , who is now VP of digital tech has presented at this conference for the last three years describing to progression of their incredible journey. Last year, Fernando presented on their response to the global pandemic, which included both cost control as well as a focus on improving their e-commerce capabilities, which has suddenly become one of their most important channels for generating revenue. This year, he will present on how top business leadership has chosen e-commerce to be a critical part of their strategy where they're choosing not just to compete but to dominate, but it is also a story that has some challenges and setbacks describing how they overcame those challenges. Fernando will be presenting with Vikalp Yadav, senior director and head of digital SRE operations and Andrea Otto, senior platform, engineer, and SRE in charge of SRE for one of the most critical parts of the Adidas commercial ecosystem. Here's Fernando, if he called and Andrea,
This is for all the optimists, all those brave enough to be optimistic, actively on not rebellious optimism. It's what allows us to see possibilities where others only see the impossible. Seeing possibilities is putting spikes on a shoe. Changing sports and minds for
Seeing possibilities is opening the door for all to play with who we are and we're going to own, it will never shut up and dribble. Let's do this together. Baby, seeing possibilities is doing something that's never been done before. And then going even further to see what's possible. Seeing possibilities is saying, why can't we leave the planet better than when we found it shaping a better future together. We're building a whole new mindset of what is possible. Seeing possibilities is taking three bold stripes and making them a cultural icon for all people. You see possibilities. That's when we're at our base, we are audio goes and we see possibilities. We others seeing possible.
Fantastic, fantastic. Impossible is nothing and welcome everyone. I'm blessed to be here for the first year in a row. And as useful, I will start with the progress report and it's a good timing because we just released our company's strategy for 2025. And this year you will see is heavily heavily into technology. Afterwards, like every year we will deep dive in one topic. And after three years talking about growth, DevOps and freedom, uh, we will tell you the story about outages control and release fitness at scale. So my name for the ones connecting this year for the first time is for . And I am since this month, um, May, 2021 heading up the data technology department for that Adidas. After six years in the company foundations working into the platform platform engineering and technology engineering expansion on talent across our tech hubs. This year, as I said, we did dive into operations at the scale.
And for this, I brought with me because the head of digital operations when necessary and Andrea, one of our super talents, I'm currently in charge of the SRE team in one of our most critical areas, we have a mobile services that contains some of our most critical and need to be stabled by the screens like content and pro data management or our checkout API. Uh, do you all know what he does? But we as a company, half a revenue of 20 million euros per year with a net income, more or less of 429 million. And we do these things to our 62,000 employees around the globe, but well, let's just start from where we're coming from. So we had, uh, four record years between 2016 and 2019 and the relentless execution of creating the new or former strategy deliver exceptional brand momentum and financial results for us.
Then 2020 came and allow us to prove our resiliency as a company in all aspects, not only in technology and prepare ourselves for the future, despite of course, unprecedented challenges coming from COVID-19. Uh, it was also a year that accelerated extractable trends in our industry, making the sporting goods market even more attractive in the future. But 2020 was especially that for technology, where we had to read them about budget by 20%, more or less while our most demanding area, the mentioned area of direct to consumer touch points. We have speed as the biggest growth in our history with 53%, uh, revenue. So this is an area moving for barely 500 million revenue, uh, in 2015 to 4 billion in 2020, and needs to go to 9 billion in 2025 being almost half of our business. It was also that a year, a year, we felt the CIO and that situation for us, all the tech management team, including myself to move towards value driven conversations with our board members, the final result, even if it was painful, you will see is in my opinion, three years of progress into one thing natural year, as I sold last year, uh, our broad domain map and our areas is the center of every decision that we do under the last year.
It was more clear than ever the differentiation between the red areas, where we win and differentiate from our competition. So this is where our change, the budget goals. This is where we build unique solutions with our more than 1,500 engineers with our extended network of partners and vendors, et cetera, blue and gray Arias. However, are areas where we compete, where we maximize, what efficiencies, where w our selection typically, uh, ranges from renting or buying software. And these help us with regards to time to market and simplifying operations. These also green, green, green Arias, and green green movement is also my personal journey. So I've been moving and I'm moving now this month from the foundations of the company to the execution and saving the future of our evil touchpoints, definitively, this is the idea where we should win as a company. This is our dotcom experiences.
This is our different mobile applications. This is our digital retail experiences, and all these is fueled through our membership strategy. And of course our data coming from consumer consumer analytics. So let's deep dive into our five-year strategy on the game that is full, full, full of technology. As you will see on the game is rude in the sport because the sport is at the end, our past, our present and our future. Um, we own the game because we have what it takes, what it takes to control our destiny in a growing and very attractive sporting goods, marketing the future. We put the consumer at the heart of everything that our people are doing, and we have three strategic focus areas that are increasing our brand credibility, elevating the experience for our, uh, consumer and customer, uh, customer and consumer, and to push the boundaries on sustainability.
Two enablers will set up up for success innovation across the entire company and the acceleration of our digital transformation throughout our entire value chain. But let's focus today only in two things. Our experiences I told you is where we win on our digital. That is how, how we are going to win there with regards to experiences, uh, by 2025, our experiences no longer think in channel. So we move beyond online and offline to own the game by embracing consumer experience as our strategic focus area, how do we do it first? We will become a member first company. We will connect with 50 500 million members through personalized experiences and amazing brand moments. 500 added as members means that one, every four wassup users across the globe is analogous member. Second, we will evolve our operating model to address consumers more directly. So we move from being a traditional traditional wholesale company into a 50% direct to consumer business.
And this is changing dramatically, our company operating model. So we come from a wholesale model where our customers or wholesalers knew our, the final consumer to ourselves, knowing the trends and adapting quickly to the trends. We come from our wholesale base two year Callender from ideation to self to store into taking real-time decision, a business or tech decisions into a hydraulic or a special sale. And these are the systems that we need to do in a second, or not even in a minute, right last but not least. Our key cities strategy keep being our PCT, keep being our global amplifier of trends. And therefore we will expand our portfolio from, uh, six key cities into, into 12. And, uh, and how to do that. We do it through, through technology and through until they get that right. And the new strategy changed dramatically. Uh, our positioning and requires tech repositioning from being at tech set up this department into a competency of value driver.
And this is a journey that luckily we started five years ago and now in tech, and thanks to this, we are on the table of decision making. So we are significantly investing in building up our, uh, our internal backups talent in the seven tech hubs that we build around the globe and already in 2021, we will hire more than 750 people increasing our engineering footprint by 50% to, and so that also with these people are thriving, uh, the highest impact they can business are integrated in operating model, driving end to end accountability into the ownership for the different products. We are also investing in the utilizing the core processes of our company. So by 2025, the vast majority of our net sales suit come from products that are created and sold in digitally to our customers. For these, we create capabilities, including our 3d creation engine, our digital go to market, or a complete, a complete re-implementation of our ERP ecosystem.
As for HANA with our with our digital first mindset, we need to live it as data and analytics. Not only like we are doing in our consumer, in our, in our digital, uh, consumer analysis, where we really use advanced analytics for our econ business, we need to use these capabilities overall from our end to end cycle from creation, manufacturing, uh, logistic, or, or of course, but let's deep dive on the people talking, right? So onboarding 750 people in seven different locations is not easy. We make all the effort for every single engineer that we are hiring to fill a critical role for the company success. So they need to feel, and you can see in the, in the left side of the screen, in this triangle, they need to feel this duality between the product and the value that they are driving. And they are bringing to the company and their capability, their technical capability, their craft, their career.
They need to feel that they have a long-term career in 90 days because they can develop their skills. But at the end, it's super important for our technical people. And all of this is empowered by our hub strategy. Our tech hubs cannot be places where we throw simply people to see the other desks, our tech house, each one of them has some model has a purpose and have a vision. And with this will help not only on hiding, but also on pretension on innovation, the sense of belonging. And non-performance two examples. So we have our festivals that the cup was the first one. What I even started working for Alita six years ago and is moving from being the development hub for the company into owning globally some of our platforms, some of our critical services. And he is going to be the hub that is taking one step forward into our cyber liability engineering capability.
And you will see later, for example, uh, other examples, the new tech cup of India, uh, where we are growing 450 people only this year is going to be the hub that is going to build or rebuild our future capabilities in.com, retail or even planning. So what happens then when you invest in technology and you move from being a service department into a competency enabler, what happens when you integrate is mostly with your different businesses? Then our business leaders are proud of us and are proud of tech. And you can see here in the screen, the example of a, of, I think that the two business leaders that are interacting the most with, with technology these days, but that is our head of digital. That basically for him, there is no, there is no barriers. There is no, there is no reporting lines. There is no organization.
So we need to be as progressive in tech and data, as we are with our digital strategy or our e-commerce strategy, we need to speak up. He's asking us to speak up or Nigel Griffith. He's the head of sales of the company. Uh, he said very bluntly. He is not going anywhere to any meeting. He's not taking any decision with uptake on the table. So my little piece of advice to the different, to other tech leaders that could be in my, in my place around the globe, invest in what you do better investing in engineering, invest in data, invest in creating architectural roadmaps with regards to your products, your team interaction models, your platforms build versus buy decisions, speak up. So the data and facts for your value stream lead and flow time, meantime to detect and recover V on the table, speak up, but please post the decisions into the pro owners into your business owners.
Don't be a service department anymore. Good, but let's stop talking a little bit about the, about the strategy and let's get into the story about volume. And yes, you can see on the screen Jesse Owens. And yes, if I using Abby Dazzlers unique technology with the spikes in his suit, he was fast, but not fast. He was the fastest, right? It's like, if you can see a smaller startup, they really kind adopt technology very fast and put it to life. But even just your wins could have never run 400 meters in less than 40 seconds by himself. For that he needed a team, he needed three other colleagues and he needed to synchronize very well. The handovers in these relay races that he was running on the 30th, you can see this as a scale-up right, or also reflecting back. It's a little bit the yearning of our leaders in our growth, in the digital ecosystem, growing from a bunch of teams to the, to the more than more than 150 200 teams that we have right now in the, in the team.
But also the reality is that growth can kill you. So imagine all these people in the screen crossing the river in the New York city marathon, if one of them accelerates or the am sort of stops 10 other might fall on the floor. And this is basically what happened to us in November, 2020. And we call our volume growth and our strategy for this growth mindset, uh, 10 X, because Y 10 X, because 10 X is more or less our growth rate over the last year. Typically we are growing, uh, almost 50% in the guitar business in revenue. But the reality is that in order to grow this, we bring between two or three more times, uh, visitors into our, into our platforms. But the reality is that we store more data. We track more data from them at the end, this is creating 10 times more technical traffic and technical load in our different platforms, but it's not only technical load on the numbers that you can see on the left side of the screen is also our number of teams and capabilities constantly growing as well.
Of course, as the dependencies among them, I'm connected with them on the next aspect as per Brooks law, right? And this is how we got to a point last November, where after years of stability, after years of growth and freedom, we go really into crisis mode. So we got, we got like four or five really, really bad outages before, uh, four or five. This is telling us that we had reached our limit and I'm things are, we, you know, have things under our control, right? And it was in the worst time ever after, already being tired after a year of COVID. And, uh, and also before, before our peak sales period. So November and December, we have cyber week. We have black Friday. We have cyber Monday, 11, 11, or 12, 12 in China. And we do almost half of our digital revenue, uh, every year in, in these two months.
And we had them with all the pain of my heart after these outages to bring three VPs in a room to approve every single change or release during that two months. And definitely if you ask me, it's not the best option by the book. And I was one of them, I was one of these three VPs in the room, and I can tell you the reality, how clueless at the end we were on some of the details. Let's not forget that we have more than 550 lines of million lines of code in our medical system. And as I told you, um, almost 2000 engineers counting, counting our partners, right? But at the end, at the end, these crisis was good because the big period was a success. And also this crisis was the beginning of a new wave of how we do operations and release management for readiness. And for that, because, and Andrea will give you some details going forward.
Thank you, Fernando. Hi, all, have a look at the slide for ones read through it. Hi, I'm and I'm part of Fernando's team currently spearheading digital ID operations and taking it to the next level. Why do you think we had three VPs personally sitting in that room in November and approving each individual change that goes into a productive environment? Yes, we had a challenge, but in this scenario, there is one part that is ingrained in our DNA, right? From the point in time, when Erin asked her that you see here made that shoe for just yoga and switch, Fernando showed earlier, the consumer experience has to be as reliable as our products, and that's the motto. That's the main element of the DNA that we drive in order to do so at Adidas, you will look at the volumes at which we are scaling. And at that mind boggling scale that we operate, it requires a 10 X mindset, which is a complete shift from what we are doing right now.
These are some of the statistics that you see from the recent past, where on a peak day, the order rates that we were achieving for affordable and chop is at 2000 orders per minute. Now, just for a second, imagine a scenario. What happens that as part of our strategy, when we want to double that order rate for those consumers, that we are trying to reach out to 168 million as of today, and which Fernando, you mentioned, we are planning to reach to a 500 million by end of 2025 are sending out 11 billion touch points every day. Imagine what would happen when we double the skill, we are driving a complete new generation of sneaker heads through our hype drop products. Imagine a scenario, we are shooting for 1.5 million hits per second. When we drop those amazing shoes on our sites. And that too, when we are doing just 200 hype, but ear today, when we plan one di one hyperop per day, what would be the scale that we are looking at?
Ultimately this leads to us deploying multiple times in a day, making changes and addressing the experience for the customer. This requires a complete shift on mindset and journey behind it. And this is a journey that I would like to share with you, how we evolve this journey from a place where we had reliable aspects of our experience and how we made that more stable and resilient. We like any other organization. When we evolved into the dev ops model, uh, we had angry birds fighting over dev and ops, but at that point in time, we really made that realization that if we want to stand out ahead of our competition, reliability is key with that realization. We adopted a bunch of ITL processes, which involve, uh, measuring aspects of stability in terms of how fast am I closing off my PV interruptions? How do I define my Bevens?
How do I have my SLA being put in place, leveraging those best practices of IDI L as a result of which we evolved ourself to a certain aspect, but still there were some challenges that were brought in and with the product led set up, which Fernando had shown earlier in one of his slides, what we also saw that an opportunity where we had the portion that we had was to implement some of the SRD practices, the site, reliability engineering practices, where we were able to answer three key questions on stability. First question, how do we detect such interruptions that happen on our productive environment as fast as possible? Second question, how do we make sure that when such interruptions happen, that we fix them at a very fast pace and last first, and the first important question of all, how do we make sure that this, these interruptions don't even land into the productive environment while answering these questions?
We adopted some of the best practices on observability in our SRE operational setup. And we scaled them across these products, through this learning. We also got some additional, which we could recognize as part of the journey. The first challenge that we saw was that individually the products were super happy. When you look at the consumer journey, right, from the point in time, you log in to the point in time, you check out, you go through a set of experience across those 53 odd products, which, which we shared earlier, and each of the product were performing fantastic when it comes to the meantime to detect meantime, to restore. But the challenge was how do we connect these experiences across in a, in a VUCA world, in a complex world? What you're seeing is that interruptions are not individually into one product, but they are connected across.
And that's where we realized that everything is connected in order to do that. There was a second realization that we also realized that we need to think about value stream as a thought process. How does it matters for us? For example, imagine an outage in a very small organization, in a very small business organization, like something in Brazil or a small market. What we observe is that the interruptions are lower in values, even though they are slightly longer in duration, vis-a-vis interruptions happening in bigger markets. Like in Europe, the interruptions are smaller in nature, but the amount of heat that we get, just because of that, meantime to restore is very high. And these two things we brought together and put ourselves into shoe off our business stakeholders. And that's where we invented the KPI of percentage revenue bleed versus next stage. So the learning of everything connected in the form of value stream and the impact of that value stream being measured with a KPI, but since revenue bleed was the net seeds ultimately led us to the key three pillars of success. While we are answering the questions, observability, the key Moto there is how do we detect interruptions before they happen, leveraging the principles of AI ops resilience. How do we make sure that we have the least amount of friction in the consumer experience and make their experience more stable and last but not the least released X release excellence? How do we ensure that our products buy the freedom of releasing new features into the environment? And this is exactly what Andrea is going to talk about in the next segment.
Thank you, Vic. Hope so. Hello everyone. I'm Andrea SRA lead in web and mobile services, as Fernando mentioned before. And now, um, our we'll walk you through the release fitness concept. So what is it about how we use it and why we need it, right? So, but first I want to just share this big picture with you. Uh, this is our weapon. We'll buy your area. As you can see, we have many different teams, many different services. And for the last couple of years, we grew a lot. So our environment became huge and the complexity and crew as well. So this is not only about now, this is not only about how my system is, but also how is the whole ecosystem? So it might be that my application will impact some other applications. Some other applications might impact me and also something in the environment itself is happening that I should be careful with.
So that's why in this big picture, you can see that there are many areas right now. That's why we need the release fitness. We need to know how is the environment, how is everything before we released? Um, we had some challenges, uh, as Fernando and also we called mentioned before. So from a couple of services in production, um, we started developing digital products at scale. So then back in November, 2020, we had three VPs in the room. Can you imagine how expensive, how time consuming, um, and how boring it could be? Definitely something was wrong. And we had to work on the release process. So there is process needed to be standardized. So all the product teams could work the same way. So that's what we did. So we worked with product teams and service management team to find what's a good set of KPIs or factors that need to be checked before any release.
And as you can see in this picture below is an Excel sheet where we have all the factors, all the checks that the team needs to do. And then it's a, self-assessment, self-assessment that at the end, you're going to have, um, it's a goal or it's a no-go, but as you can see, it's manual, right? And we have that amount of product teams and we want to be a jail. So we want you to deploy as much as we can a possible daily, if possible, multiple times daily. And can you imagine how the teams reacted when they received the spreadsheet? And they were told that they needed to, to feel that every time before a release, of course it was not the best reception. So that's why release fitness came into place. What if we can automate all those checks and the, and we have a single signal go or no go.
So we can remove all the communication, all the manual process and try to get all this information, all those KPIs and factors we can bring them automatically. And it's exactly what we did. It's exactly the release fitness concept. It's a release based on KPIs. What we have is, um, a unique signal based on a set of KPIs. And we see that from three different angles. So we have the system level. That's how my product is. We have the value stream. So value stream, uh, one example where Adidas is the checkout flow. For instance, for checkout, you have different product teams working for the whole value stream. So we have the front end, we have checkout API. It's where I am. Particularly we have another backend service, we have payments. So we have the whole value stream and other dependencies, of course. And we also have the environment.
So for the environment where you have all the platforms, we have all the events that might be happening during that time. So a combination of those three will give me go or no go, it's the same analogy we can, we can do an analogy with our bonus in company bonus. For instance, we have how I am doing, how my team is doing. We also have how my company is doing, and we also have how the industry is going. So then a combination of those three different angles we will give you, uh, were grievous, um, when one single, uh, go on a boat. And for that we have on product level. So on product system level, what are we measuring? It's not only one KPI, but it is a set of KPIs. So we have the error budget, and her budget is one KPI that we see that's very well known in this community.
So it's basically defined it with the service level objects and service level indicators. For instance, if we cannot take availability as an indicator, and depending on the criticality of my system, I can define that. I want to have, for example, availability of nine, nine, 9%, it's okay for my service. So it means that I have 0.1 of, um, unavailability expected and availability, which is okay to be an available for 0.1%. And with his or her budget, it's a very, uh, special, uh, measurement that we can go from the product level, to the whole value stream. And then we have CIC D for instance, how is my application code? How are my quality gates in the coat? We have QA KPIs, for instance, functional testing coverage. The fact that section person that we ha we also have blocker issues. Is there any blocker, anything that the QA team found that will block me to release?
So everything is automated and nobody needs to check with different teams with different tech, uh, with different tools. So it's all in the same dashboard. We have the value stream, as I explained it before. So we have different systems, downstreams UpSpring systems that might depend on me or that I depend on. So they are, they all need to be checked and for the environment as well for the environment, we have two key aspects. So we have the platforms. So the technical platforms is my Kubernetes, cluster health is my drinking's health and all the other, maybe some service that I use from AWS is it all up and running so I can use, and we have one very important in particular for, for Adidas. It's the release of the day. As, uh, as of now, you already know that our environment is very big and it might be that some event that's happening, my impact, my application.
So for instance, there is one event. We call it the hype sales and hype sales during hype sales. Um, our system is expected to receive lots of load. So it's one time that we don't want to add another variable April to that. So we need to make sure that nothing will change during that time. So all this communication is automated and they automate go, is that we have the single, sign-up the single signal that is validated during the promote pipeline. So before I release anything to production, I have this check automatically done. And if everything is fine, it goes production. If it's not fine, we will have to stop. We blocked the Butte and then the team that is responsible for the release, we will check the dashboard and we see exactly what's going on. It might be, there is an event going on. It might be that I don't have her budget. Oh, it might be anything else. So next steps is to onboard all the teams and have the whole value stream regulated with a budget with that. I can hand over to Fernando for the closing part.
Thanks a lot. so this is fantastic. And definitely taking us to the next level. I personally love systems that self adjust and self-regulate like in this case, so on one hand, we have streak release guidelines. And on the other hand, having automated FISMA checks and error budgets, telling every engineer, if they can or cannot deploy. And this is especially important on systems, as big as our ecosystem with more than 1,500 developers putting code into production. And I just want to end with a reflection that gene Kim asked me the other day, right? So why added this keeps coming every year to the devil's center price. If we are really at the forefront. So for me, it's definitely reflecting. It's helping me to reflect the year successes that year progress that you have failures and also learning by teaching others, the things that we do well, but it's definitely learning from others, things that they do better.
There's always one, there's always people doing things better than you listen. Let's accept that, right? It's especially also being close to the community. People like Jason caulks, you're not tiny smart, Matthew Skelton weakest. And so our colleagues from Jaguar, from BMW learning from them. So it's not only on the DevOps enterprise. So whenever we have a question, we reach each other and it's really a very, very enriching experience, always. And, uh, new people, new companies, the smallest startups, small digital companies, or big legacy enterprises are reaching you after watching your talk and asking you for one topic or the other. And last but not least on top, I also get access and have the luxury of being an early reviewer of some of the nicest book or books of evolution, which which are invested. So in essence, I will be here next year again, and I hope that in person, in a big venue and not remotely, because I hope that this city by Baidu's disappeared from our lives and we can go together in a community. Face-to-face thank you.