00:00:00

<silence>

00:00:13

Good afternoon everyone. Uh, by show of applause, how has the day been for you? Hey, fantastic.

00:00:24

Alright, so, so the first speakers of the afternoon are from Deutsche Telecom. Asha Sheik is a DevOps engineer, and ORs th niche is a senior chapter coach and vice president. So, Deutsche Telecom is the largest provider of broadband and mobile in Europe. And to make this happen, they require nearly 10,000 technologists. So I met Asha at the DevOps Enterprise Summit last year in Las Vegas, uh, while in line at the Starbucks the night, uh, the morning before the conference began. So I got to meet him and his team and learn about the amazing work that they've been doing, uh, inside, uh, their organization. And I was delighted to hear how he was so influenced by the talk that was given from the team from Virgin Media oh two in the prior year. So one of the reasons I'm excited for this talk is how ASHA and ORs have been able to, uh, ch start changing the ways of working in a company that is famously conservative work culture and is engaging scores of teams across the organization to improve their ability to deliver value to their customers. So here's Asha and ORs.

00:01:35

Thank you. Alright. Hello. It's great to be here. Today, we will share you our story, our story about changing the minds of about 10,000 people at a large German corporation, DOCHA Telecom. My name is Urgh. I work as the leader of the IT operations and DevSecOps chapters, and joined by Sasha, who is our tireless, uh, DevOps evangelist at Deutche Telecom. Thank you. So if you think about DT Deutsche Telecom, you would probably think about German telecommunications company, right? And yeah, well, in terms it's correct, but we are also much more than that by now. So it's much more than Germany. Of course. We are represented in many of the European countries and also in the us and it's much more than a telco. We define ourselves as a technology company. And so of course, transforming into a technology company has been majorly contributed by the internal IT of Deutsche Telecom.

00:02:45

That's where we work. And so if you just look at our numbers, who we are, and so you would, you know, see quite impressive numbers. Uh, I always find it practical to explain what we do by just explaining what happens if we mess up. So if we mess up, you walk into a shop in Germany and you can't buy the new iPhone, or if we mess up the technician who would go to your house to connect your fiber connection, can do his work, or if he mess up, then you just can't contact our service desk or call agents. So as you would, or I mean fundamentally, we can just, uh, kill the basic communication services too. So, uh, you understand, uh, it's a lot of, uh, really critical systems that we develop and operate.

00:03:38

So our journey begins, uh, in 2017. And in 2017 it began with this picture. We drew this picture in order to explain our ambitions to our own staff and our, to our leadership. And this was an era where we had, you know, everything OnPrem, uh, big monoliths. Uh, there was, uh, basically no sign of DevOps yet. And in our organization, we have set ourself a couple of goals. And of course you will find the usual ones like, we want to move to CICD, we want to decouple, we want to move to microservices. Uh, we want to have, you know, stability and scalability, uh, by leveraging cloud providers. And we of course would like to have an organization that is flexible enough that supports everything. And so besides all this, for us, probably the most important part was that part in the middle, which is the culture.

00:04:39

And we wanted to have people working for Deutsche Telecom it, who would own what they built, who would share what they built, who would be proud of their metrics because they know where they would like to develop themselves, and we wanted to move into a fully DevOps minded, uh, you know, bunch of people. And, you know, are we there yet? No, we are not there. And we have made a lot of mistakes on the way, but I think, uh, looking at what we have pulled off in an organization this size and in a, in a legacy culture like Deutsche Telecom, I think it's really interesting for some, uh, other companies too. So on the, how, how we did that, I hand over to Sasha.

00:05:24

Thank you very much, Josh.

00:05:28

Now, uh, first of all, it's really great to be here and to share our experiences with this community because there's a lot of the things we came up with. I sort of got the idea from exactly this community. So it's great to finally be able to give something back. As you all know, uh, in a DevOps transformation, just a small part is about the technical practices. The much bigger issue is changing the mindset and changing the culture of the company. Here we may have an even bigger challenge than others because as you heard, we're mainly a German company, which means we have a very German work culture, and that does tend to bring difficulties with it. As Germans, we love accuracy, we love being on time, we love clear responsibilities, which is great if we talk about German engineering, just think of the nice cars we're able to build with that mindset. But if you're talking about changing something as fundamental as culture in an IT company, then things start to get tricky. We're just not great with change. We love consistency. Uh, for us it's normal to start your career in one company and then decades later go into retirement from that same company. I mean, I'm often referred to as one of the young wild ones, and I've been at telecom since 1998, <laugh>.

00:06:55

So we needed to come up with effective strategies on different levels and these grew over time, um, to make sure that we get this culture change going. And I'd like to show you what we came up with.

00:07:08

As Arris mentioned in 2017, we started with our agile transformation by setting, by deciding to do we would do safe as an implementation to do our agile and DevOps ways of working to figure out how to do that effectively. In 2018, we started and set up the first, so-called hubs, which are basically a release train according to safe, which we choose, chose to reflect what we would encounter in the company. Overall with these seven hubs in the pilot phase, uh, we wanted to learn how to do and scale effectively. The hub I was involved in was responsible for creating new products for our business customers. And from the get go, we had a couple of advantages on our side. First of all, um, we had really good starting conditions. Um, we were mainly in a, in greenfield environment, which meant we didn't have to take care of all those crazy monolithic legacy applications. At the same time, these pilots attracted young people, so the spirit was good in that team as well. And we had a very high level of management attention on these first seven hubs, which really helped us if we ran into problems to solve them quickly.

00:08:25

Of course, we also had our difficulties in that time. Um, we were still surrounded by legacy processes all over and they couldn't keep up with the increasing speed we had in those pilot hubs. Then we really didn't have an understanding of what DevOps is yet. If you asked at that time someone what you, what DevOps meant for him, if you had 10 people, you usually got 12 different answers and 15 reasons why it's a bad idea in the first place. And we didn't have any central CICD tools. Uh, so each of those hubs had to figure out how to do, how, where to get a GitLab from for themselves, for example.

00:09:01

But what we did have in those hubs, at least in the hub I was in, was a system team. Uh, and I was in the system team. I set up the tool chain for this hub, and then I started teaching the people in the hub how to use this tool chain. So basically how to do the technical part of DevOps. And that's where I really fell in love with the topic because I could immediately see what advantages DevOps brings, not only to the company, but to the individuals and the teams as well. But do remember, I was still referred to as one of the wild ones in 2000, um, 19, we really started scaling. So we were satisfied with the results of the pilots and started really rolling out more and more of the hubs. One of the things we had learned in the pilot was, of course we would need essential tool chain. Uh, so one of the first hubs we created was the, so-called CICD hub, which provided exactly that tool chain. And one of the teams in that hub also offered a service, which was teaching the people how to use this tool chain again. And because I already loved talking about DevOps so much, I moved into that hub and started telling even more people about DevOps.

00:10:12

What we quickly learned in those sessions though, was at the end in the q and a part, people didn't ask about technical stuff. They didn't ask, how can I add another stage to the pipeline? How can I automate this? What they really asked about were things like, um, mindset and culture questions. Why do we need it in the first place? And that's where the German engineering started to get in our way. They wanted to know who would tell them it's okay to deploy, uh, or who, who, who gives me the approval to put something on production and who's responsible? What do you mean team? Which one of us, we need a name? Yeah. So we really needed to figure something out and, and do that. Fortunately at that time, I was already starting to get involved in this community and I had heard people like Mick Katon or Mark Schwartz talk and knew that DevOps is not just about the tools, it's more about the mindset. That's why we said, okay, we need to focus on that and do something dedicated for the mindset part. And that's why we set up the DevOps coaching team. I got to be a product manager for that team, and we set up a couple of services to help us.

00:11:20

The first service we set up was the DevOps Health Radar sessions. DevOps Health Radar is basically a tool from safe, which goes through 16 questions over the whole development lifecycle of a product, um, and where you can see how the team can improve in those areas. But the BA added value we brought was, we did for our sessions, um, going with the team through these questions and getting the teams to talk to each other inside the hub and getting the people in the team also to talk to each other. And that was really valuable. People loved that and kept booking that. We really had to figure out how to do all those health radars. Another ser service we did of course and offered was DevOps introduction sessions to get down from the 12 different opinions to maybe five. Um, and they were also hugely popular. Suddenly everyone knew okay, asked the bearded guy about DevOps. And I must have done about 50 to 60 of those sessions to sometimes hundreds of people. And, but I think the most important service this team still offers today, um, is team consulting. So we went out, we talked to the teams and we helped them figure out what their next steps would be in their DevOps transformation. And then of course, made sure they also put it in the backlog.

00:12:42

In 2021, we decided, okay, we'll add another layer of complexity to our overall transformation by introducing skill-based chapters, which meant every employee was assigned to one of roughly 20 chapters based on his or her primary skills. Now, the vision behind this chapter approach was that the chapters would drive the individual topics into the company and also towards the people. So I'm, for example, in the DevSecOps chapter, which means we are of course responsible for driving the topic of DevOps into all the hubs To be able to do that, we said, okay, every people lead has to be what we all, what we call a Spock or a single point of contact for that topic. And each one of the people leads has roughly six to eight hubs he's assigned to. That gives the, the hub a dedicated person to talk to when he has has questions about DevOps, which there are a lot of. And at the same time, it also gives the chapter organization a, a possibility to address topics around DevOps to the hubs. So again, the services this box and the chapter provides is of course, demand management, which we try to do in a collaborative fashion, not just bring me five DevOps engineers, but more of what do you need? Maybe you, you be better off with two DevOps engineers who know how automate, how to automate better and skill management, of course. Um, so we need to, by talking to the hubs, we need to figure out what kind of skills they need and then make sure that we as a chapter can provide those.

00:14:16

But again, I would say the most important thing that we do is, um, helping the hubs, uh, to figure out how to do DevOps. So really talking to the hubs, um, we make sure that they have enabler stories in their backlogs that they, we give them impulses to think about and of again, that we make sure that they really put that in the backlog. So now we have the teams covered by the system teams, uh, by the DevOps coaching team, but also by, um, and we have the, the hubs covered by this box. So the, the management of the hubs especially, but there's still one area missing, um, where they could also sometimes use help, which is of course leadership. Um, in 2022 was a really tough year, as you all know. There were a couple of external influences which really impacted our ability to complete the transformation. One of them, of course, being the war in Ukraine, it made it necessary for us to relocate a significant number of our workforce, over a thousand people, out of Russia as quickly as possible. And these were people working on very innovative topics, which were very important to our business as well. Um, and we didn't have the skills or the capacities in other countries to just take that over.

00:15:38

So what happens in difficult situations, of course people tend to go back to the fall, back to their old ways of working, uh, and to their proven habits. And one of the first thing that goes over the table, of course, is changing much, especially something as intangible as mindset or culture. Now, I was already known as the face, or you could say the beard of DevOps in our company, and many people were already listening to me, but how could I also get the leadership team to listen to me? How could I make sure that I could inspire them as well to come up with new ideas and try out things which I'd heard, for example, out of this community. Fortunately, a couple of colleagues from Budapest had the same feeling that we wanted to keep pushing our DevOps transformation, and they were also just as frustrated as me that DevOps kept falling off the table. Um, and that's why we did a pitch to the IT leadership team and said, okay guys, we need the DevOps expert council, and I'm really happy to say they listened, they understood what we wanted to do with it, and they, they supported the idea and said, sure, go ahead staff that.

00:16:53

So we made sure that this DevOps, um, DevOps expert council is, um, of course strongly connected to the IT leadership team, which you can see because S is here as well. Um, but also we made sure that it's connected to the chapter and to the hub organizations as well because we really need a varied input for our team. And again, there are a couple of services we provide. One of them being we are a change agent for all the DevOps initiatives we have out there. They usually talk to me about it anyway. So why not streamline that, put them together and make sure that they know of each other and so on.

00:17:30

Then what we also take care of is of course, making sure that we have a constant external exchange with other companies to learn about DevOps. One of the things we did, for example last year was we set up a joint luft tanza Telecom DevOps Day, where we got 25 people from us and 25 people from Lufthansa together for three days to talk about DevOps. And that's, that was really cool. We also do ex regular exchanges with companies like hamus. We've talked to Siemens, we've talked to the European Space Agency, which is not as cool as NASA we heard yesterday, but still. Um, so we talked to a lot of external companies to get input from them as well. At the moment, we're setting up an SE pilot to figure out how to implement site reliability engineering and if that would fit in an organization like ours with all the Germans. Um, but the first service we set up, which is, is also the one I'm most excited about, is we said we wanted to have a DevOps maturity score.

00:18:29

So the DevOps maturity score, the idea behind it is that we needed really an overview of where we are with our transformation. I had a gut feeling because I had talked to so many people, but we really needed to have something measurable. And it was not just enough to count pipelines or use Dora metrics or something because we had so many legacy still, we still have so many legacy systems and legacy applications. They will never probably really have a pipeline in GitLab, so we wouldn't count them, but of course they still need the DevOps mindset.

00:19:00

So we came up with this questionnaire, it's seven simple questions, um, which all touch on different areas which are part of a successful DevOps transformation like culture, collaboration, uh, automation, business interaction or metrics. Each of those seven answers, uh, each of those seven questions, sorry, has five predefined answers, which on the one hand show immediately show you the way of where you should be going and what you should be thinking of in that area. This makes it easier for the hub owners themselves, uh, to fill out this questionnaire and come to the score without having to ask any nerdy bearded DevOps engineers to do it, which we hope will then raise the raise, sorry, raise the awareness of what they really should be thinking about and what they should be prioritizing in their backlog. At the end, you get a score between zero to five, which maps you to one of five maturity levels, and I'm happy to say we started the data collection around two months ago. Up until now, 47 of our 127 hubs have responded, and our over or score at the moment is 2.9 over those hubs, which puts us in the, uh, defined stage shortly before measured.

00:20:15

So this is great, apart from confirming my gut feeling, um, that we were on a good way, we can now have something to measure and now we can come up with ideas how to improve that score even further and repeat the measurement to make sure that we're moving in the right direction. So I'm really happy that we have this overview.

00:20:34

Now that brings me to my ask to, to this community. As you saw, I drew inspiration and ideas from all of you for the setup we have now of the teams and services we provide for changing towards DevOps, but there of course, is one part really missing, which is kind of an important part, which is the business side. Um, so as you know, transformation is just not, not just a thing of it, business has to be on board. So we are really looking for inspiration on how to do that. So if you have a similar organizational setup as we do, uh, if you maybe have your business side aware of your DevOps transformation, uh, maybe you have a dedicated team setup for that, uh, then please do reach out to us and let us know. We could of course keep sending the book from Mark Schwartz, the seat at the table anonymously to the business side, but there has to be a better way for that. With that, I want to thank you. I'm very happy that I could give something back to this community. I'd love to hear if this inspires you to set up teams in your company. So please reach out to me over LinkedIn or email or just talk to one of the people with the Magento College stuff. Thank you very much.

00:21:50

Thank you.