00:00:13

Hello everyone. And thanks for taking the time to watch our presentation for DevOps enterprise summit, 2020 Las Vegas virtual. We're so excited to tell your story. My name is Roman pickle and for the last two and a half years, I've been a technical project manager at electro pit in the advanced driver assistance systems. To me after being a process manager at the Austrian parcel service and being the CTO of a medium sized company, which was sold to a major industry player, I decided to escape the startup roller coaster and look for position by there. I have more knobs to turn. I'm not working on hardware and software for data logging and replay as well as hardware in the loop simulation solutions lately. I've also joined Electra bids, continuous improvement initiative. As a continuous improvement agent. I have a background in software engineering, business administration and computer and electronics engineering, C I C D and dev ops is really the sweet spot for me. As I laugh, all the things that I learned in my production management and operational research courses are nowadays applied in the it domain last but not least, I'm here to learn, which means that I'm really looking forward to the questions and discussions afterwards, especially if your opinions and experience differ to mine, you can capture us in slack throughout the presentation. If you have any questions today, I'm joined by Manya Loza. Hi Maria.

00:01:36

Hi. Thank you all, man. And hello everybody. I'm also really excited to be here and really looking forward to learn a lot from the presentations and from your questions, of course I'm at, but in the role of the continuous improvement team lead. So omen mentioned that he's in this continuous improvement program involved. My role is to lead this improvement. Actually also my role is to be the head of incubators and demonstrators. So this is very try to do something new where we try to figure out where electric it has to go in the future. So both these roles go well together. And I'm here today to tell you something about the bigger frame, that our story of really moving faster, how visualizing flow changed the baby work is embedded in. So what is this patient that we are in right now in the automotive market?

00:02:33

I'm pretty sure you have heard that we are facing disruptive change, right? So automotive industry is really undergoing a big, big change right now. So we are moving from combustion engines to electrical vehicles. One consequence being that less parts in the cars are needed. So of course also there's less to supply electrical particularly is in the software area. So these apply software to automotive players. So for us, actually, this change is a really big chance as software is getting more and more important in the cars, but also for us at the same time, that's a big challenge. So last week I heard a number of which really alarm me. So in the next five years, from what we foresee right now, there will be one complete year of car production lost two to Corona due to the changing economical situation. So this means that every year, about 20%, less cars than usually assumed will be produced.

00:03:39

So of course this will also have an effect on all the companies connected and all the companies supplying to the industry. So what we try to deal with the situation is to drive technology further. So to, to produce or to supply technology that will really help the automotive industry in the future. And at the same time, we'll have to improve the way we work. So this is why we call this whole initiative, continuous improvement, but let me start with the technology part first. So the areas we are in automated driving, so really hot topic, vehicles, structures, or steering the car. So the car is basically now a computer on wheels, right? It's an IOT device. So we deal with topics like how can we steer these cars? How can the infrastructure work? Connected vehicle is a really big topic. So I guess most of you, if not, all of you have a smartphone and this thing is so connected and it's, well, it's a smartphone, so it's so smart and you expect it to update once something is broken or once there is a safety or security issue.

00:04:47

So same thing goes for cars. So we want to make sure that cars are connected and get these capabilities. And of course, all this is really closely connected to user experience. Because if we go back to the smartphone example, you really expect a lot right of these devices, how they are handled and so forth. So we have to take this to cars as well. And of course, this is not a European topic or a, an American topic. This is a worldwide topic. This is why we have locations all over the world. So in Asia, U S and Europe as well, our headquarters is in Europe and electric. It has about 3,400 employees. And in the different regions in the world, you see different ways of handling the change, different ways of dealing with the disruption in the automotive industry. But I think all of them, all the different, um, put, uses all OEMs have in common that they see a strong move towards software and they see a strong urge to improve all the time.

00:05:53

So we started our improvement adventure last year, some what we call timeframe initiatives. So we had certain programs for instance, called delivery performance. So we knew we had to improve our delivery performance. It did not take long for us to notice that it does not make sense to have something that's restricted in time. And that people try to do well basically next to their daily business. So we had a group of people working on this topic that had completely different jobs. And while they had a timeframe to get the topic done, then again, when they started working on the topic, they did not even know what it would require in the future. So that's where we moved to the continuous improvement. What we want to achieve with the continuous improvement is the from customer projects to products view. And of course the name of this slide or title of this slide is no coincidence.

00:06:54

Of course, we were really inspired also by me, Kirsten's book on the topic. So what does this mean for us? ? Well, traditionally is from our history. We are really a project company. We work for big OEMs. We supply what they ask us to supply, but of course this view does not really scale. So we were producing really specific solutions or specific projects for an OEM. And then we had a really hard time to translate this and to sell this to other OEMs. Of course, this approach does not really scale well. We want to deliver value to all customers that we could potentially have in the market. And that helped us with our strategy. Another thing that we realized in the last let's say a year is that we need to have a stronger focus on optimizing the overall value stream. So we often optimize locally, right?

00:07:50

So there's teams finding out that there's something in the process that could be improved. Then again, if the team is improving something, it's moving faster now, but it's still building the wrong feature. This will not create a whole lot of value for the customers. So this is also a perspective that we are trying to take much more strongly now to look at these whole value streams. Of course, since we started this journey of improvement, there's really some challenges that are becoming more and more visible. So for us as a company, you might be hard to imagine, but electric worked for nearly 30 years without having really a pronounced division that people were working towards. Why did this work well? Because we remained in serving our customers in projects and they told us what to do anyway. So we did not put a strong focus on developing our own vision.

00:08:45

And really it's a challenge for us to learn this also, this is reflected in contradicting goals. So if you are a project company, one of the strong KPIs is utilization. So we make sure that people are working, that they are busy all the time. Are they working on the right things yet? Another question. So this is something that we still really struggled with and also strong, strong focus on daily business is something that's really inherent in our company culture and that we are struggling with because we need to take more time to reflect and to learn. And of course, a lot of these things can be achieved also with dev ops to deal with these problems. So as you see in this red circle, this is like an observation place. So assume that this is management looking down at the big valley. So what is really important for us to learn now is that the things are happening in the valley.

00:09:44

And actually, so these are the, the arrows in the valley, and this is, uh, the flashes. So this is really where things happen. This is where people are, who understand what's going on and where people are, who can come up with strategies of improving things. And actually if you should be getting away from monitoring from above and telling people what to do, but we need to listen to the stories that are told in the organization. And we need to enable that these stories can be spread and people can learn from them. So I now want to give back to a woman because he's actually, one of the people are really active in our company creating such stories. And he'll tell you his story now.

00:10:26

Okay. Thank you. Manya. So when I turned the lecture a bit in 2018, the main business problem we faced was delivering value in a flexible way at speed, high quality to intern excellent customers, um, the full cycle of building and testing. One of our software and hardware products took more than 24 hours. So when you did something in the afternoon, we sometimes didn't get feedback until the following day, but the day after it had a negative impact on developer moral, and it felt really like quick sense, the more we fought it, the more it pulled us in, we knew there must be a better way. So applying the three ways of dev ops, especially by experimentation and by identifying bottlenecks in the build and test run, we were able to cut the full build test cycle by effect of three in the first few months of 2018, moving coat to a Monterey pool and containerizing our built environment in 2019, allowed us to provide feedback to our developers on every commit within minutes, not hours.

00:11:28

Furthermore, automating our delivery, allowed us to provide a new version of our software with the click of a button. This was great, and we felt happier and so freaking at job, but after putting in so many hours and improving the deployment pipeline, investing in automation and deploying new technologies, it was time to ask a fundamental question. Are we really moving faster? So one aspect that I really liked about my job in the operations department of the Austrian parcel service back in 2009 was the fast physical feedback and visibility of problems. There were more subtle and hard to find process areas, of course, but if one of the main systems or processes did not work as expected boxes started to pile up at a bottleneck, providing a hard to ignore indicator for problem. I moved on up after about one and a half year, but since then, I always missed this clear feedback signal in my it jobs.

00:12:27

I was missing ambient awareness. I think I first read about this concept in Michael Nygard's, 2007 book release it. The idea is to create an ambient display, an interface between people and digital information, which represents data. For example, the health of a system with the help of sound visuals movement, or other cues, the virus, various ideas out there from simple displays to ambient lamps, novel lamps, a PLM USB, rocket lamp launches, traffic lights, you name it. These kinds of information. Radiators should be put in a highly visible location to remote, to promote responsibility in the team. They showed that you have nothing to hide. They provoke conversation, and they can be traced back to you may have already guessed it. The Toyota production system, uh, Steve Poole from IBM held an inspiring keynote about their sports and culture Def Europe, 2018. He told a story about how sharing insights on the dashboard, closest communication gaps, forces discussion on how to generate accurate data and change your culture.

00:13:36

Putting data on the dashboard makes the problem real again, before that it's just data in a spreadsheet. So I had already collected the data that I wanted to show, uh, in the Wiki for a weekly status meeting, but I wanted to collect them automatically and have them up to date all the time. So what I really wanted was, uh, an automated dashboard. What is more, I remember the quote from Winston Churchill. So we shape our buildings and afterwards our buildings shape us. It also reminded me of a Skoda plant and the w project house, uh, discussed in trauma. Thomas J. Allen and counter hand spoke the organization and architecture of innovation, um, where the employees have to pass certain points of the assembly line or up to date prototypes before arriving at their workplace. So working in a dispute team, I wanted to have the data and available on our intranet, but also on a highly visible screen in the entrance area.

00:14:38

Everyone passes by a few days, times a day. So I already had a raspberry PI at hand. Uh, but I quickly learned that getting my private device and the company network was of course impossible. What was even more startling for me was that it was very difficult to get an additional monitor, to show the dashboard. I asked for it every other month, but given that we were growing, there was always a scarcity of time. Also my own and resources. In retrospect, I was very successful last year with piggy begging a plan change. So part of our company moved from one office to a new office down the street. The new office was renovated and the visualist was created. So I asked it stuff, uh, that I need, uh, the staff for my dashboard, basically the raspberry PI and the monitor. And I think that given that we're in a change mode of solving problems, uh, buying hardware and setting up the network, it was easier to get this stuff approved.

00:15:35

And I got these devices. There were also some logistical problems, which I guess is normal, uh, with setting up the new office space and the some time to play around with a framework called smashing. So based on the metrics I had collected by hand for a few months, I wanted to visualize the following things, the next milestones and the important dates of releases, open pull requests and their age, uh, open support tickets, chiro tickets per status as also visible on our Kanban board and the status of Jenkins shops, including build time and failing tests. So I was ready to put some time into implementing a first version of the dashboard and I set it up in the hallway ready for the official opening of the office. The dashboard sparked a lot of interesting discussion during the opening party and ever since the dashboard has been part of the new office and evolving into an important indicator of the current status of the product and project and the source of new change initiatives.

00:16:35

I had succeeded in bringing back and be the oneness that awareness, but that's when I noticed the problem. There's a scene in Aliah M Goldratt's classic book. The goal that I read after listening to the, beyond the Phoenix project, audio book. So Alex, the leading character of the story is very proud of the increased productivity. Uh, they get in the plant by applying robots when Shona the management query girl asked him some questions. So the summary, uh, in summary, the dialogue evolves, something like that. So is the company now making more money? No. Did you ship even one more product? No. Our plant inventories down. No, I employ expenses down. No. And then she'll assess, then you didn't really increase productivity. Your inventories are going through the roof aren't they? So looking at the dashboard, the inventory was staring me in my face. Imagining all these tickets were boxes lying around in the hallway.

00:17:34

They would have been way harder to ignore. They don't have any value as long as they are not released. Furthermore, it doesn't really make sense to add more. The bottleneck in development had shifted to testing and they were creating a lot of however, when I read about the typical progressions of bottlenecks in the dev ops handbook, I was reassured that our efforts were going in the right direction. It also reminded me of the checker of automation mentioned in 2018 state of the DevOps report, which states that you really need to dis relentless improvement, refactoring, and innovation to reach a state of excellence. At this time, I had just finished reading the project to product book by McKesson, looking at our current state, I was especially interested in the throughput part and aim to measure flow as defined in the book. So I aim to measure flow load flow, time flow, velocity, flow efficiency, and also wanted to visualize flow distribution.

00:18:36

I also had to look at tracking business value, cost, quality, and team happiness, which we do with the survey quarterly and correlated to the flow metrics. However, I quickly noticed that it's really hard to get some of this data, especially in an automated fashion. So I created another dashboard that visualize this. These metrics notice that Vida, it provides some insights into features and defects. We currently do not track risks and technical debt. Some are in the improvement category. Every 60 seconds. We rotate through this flow metrics, dashboard and the status dashboard. After the next deployment, I stood in front of the dashboard in the hallway and it dawned on me. We were shipping more often, but as we didn't deploy from master, but the RAV4 patches from a release branch on average, we got slower. We had the fast lane for fixes, which were fixed on master and back ported to the release branch, but it still took us too long to ship features, which were waiting to be released.

00:19:37

So we were looking into cutting our release cycle for major releases from every half year to each quarter, even more often still, it seemed as if we were always late with priorities and requirements changing in between these cycles. And it felt like we were improving our development process, constantly running, but remaining in the same spot as in the red Queens race in Lewis, Carroll's through the looking glass and what Ellis found their book well in our country set out is still paint, painting a little. You generally get to somewhere else. If you run very fast for a long time, as we've been doing as slow sort of countries that the cream now here, you see, it takes all the running you can do to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that, as I later found out, we were trapped in the local optimization and urgency paradox described, but Jonathan smart, valuable ideas sit 12 months to 18 months, uh, in front of, in big front plant, big upfront planning phases with no sense of urgency, no questioning.

00:20:47

They might add more value than what has already been locked in the plans for the year. And as soon as they reach the product development team, they are actioned. So I had already heard of similar phenomenon called waterscrumfall. Um, but this, uh, via, so fricking agile, uh, picture from class Leopold, uh, really drove it home for me. So by the way, if you're interested into the deck in the dashboard, you can find the code on GitHub and a blog post on my website, if some more technical details. However, after implemented the dashboard, I found that there are now several professional and open source solutions that cater for the same or similar problem. So if you have a more complex setup or want to do something more serious, you might want to look into these. If you want to know, more Forrester has a very recently updated report and value stream management solutions that might be of interest.

00:21:42

So we found that we had a limited system focus and we needed buy in to influence the process up and downstream. Luckily at the same time, the organization identified focus programs to improve flow. And based on these started this continuous improvement initiative that manual talked about, we were able to connect to the program and harness what we learned to try for, for change. For example, the moved to a lightweight quarterly planning cycle as discussed in Gary group, his book, a practical approach to large scale agile development to solve changing priorities and the urgency paradox. We now track working progress more closely. We work in smaller batches and we double our release frequency in 2020, providing monthly patch releases and quarterly minor releases. We are heavily investing in test automation and we built a new test deck as well as simulation and emulation capabilities to test more of our use cases automatically. And last but not least, we are also discussing about reorganizing our teams based on the cognitive capacity as discussed in nephew scalpel and manual pay spoke team topologies as the cognitive cognitive load of our team, uh, was too high, uh, while we made considerable progress in our journey, challenges still remain so Mangia. Now we'll tell you more about the road ahead.

00:23:06

Thank you. So I guess it's basically always about finding the balance between all these activities, activities that are going on in the teams. So like the story that Alma just told you and about bringing this to the company level. So, which is really more of what my job is. So what are the challenges that we are facing in the future? What are the next steps? There's some steps that are quite concrete. So as Homeland pointed out with the image of a clause Leopold, if you optimize locally, you might get faster, but not better in a way. So what's really on our list of things to do is this cross product prior to station and planning, we need to have good way also across our products to find out whether we are doing the right things and better. We are doing the most important things first.

00:24:01

So this is really something that where we need to improve. We need to expect tablets, meaningful KPIs. And of course the flow framework is a good inspiration here. And also this is part of our approach. Try out in places, try to find out what works, what makes sense for us and our business context. What really tells us something about performance and then put this on the whole organization, because of course, we also need to compare between products between departments, but this is really the second step. So first we need to find the things that are meaningful for us in certain circumstances be, are conducting value stream analysis. So not only for this product, but then for our products, which also then helps us to find the problems, identify the next, try to determine what the biggest problems are and how we can move forward. But also again, to show, uh, what is there going on between our products?

00:24:59

Um, maybe what's more valuable for us as a company and for the customer of course, and what isn't. So this is also why we talk a lot about these cross product topics. So we identified some products, some topics that can only be solved on a company level. So for instance, budgeting is one of the best examples. If you want to work in this way, that'd be just described or in a more if ops oriented way, we need to rethink the way we do budgeting, but this is something that can't be done on a department level, of course, but that we need to address on the whole company level. And we need to scale continuous improvement, of course, because if we have these stories, we need to take them to the company. We need to ensure that other people are learning from it, are adding to it.

00:25:46

And that we, as a company can move forward. Right now, we are working with the CIO team and then the so-called continuous improvement agents in the several products. Uh, this is the set up that we currently have, um, and very try to bring topics to at various departments and to scale. So this is well rather concrete, if not very concrete at all. There's also some more abstract topics on the next slide. Yes, there's some apps like in the sense of like really big challenges that remain making our achievements relevant and sustainable for the organization. It's one of these topics. So for example, of you're often asked by how many minutes, hours and so forth, did you improve your build times? And then we can say, well, it's also his whole mindset. We improve them by a factor of all by 90%, right? But then we need to ensure that the next time we don't start in the same place and have to improve for another 90%.

00:26:53

Again, we need to make sure that we learn what we did wrong in the first place to make this results relevant. You need to learn from this and maybe next time even start better than we did in the current situation. And actually a lot about continuous improvement and also deeper for me is removing impediments that are rising from our company's structure and from our culture. And I'm certain, you know, this, there are certain ways of doing things here and these ways are not always welcoming Miller coming in new way of working, of course, because people's positions are at stake. Everything needs to be discussed. So this is really part of the hard struggles that we have here, and we need to reflect and to learn about our organization, which sometimes it's really hard because we are part of this organization. So it's really kind of hard to tell where the problems are, but we need to train ourselves to get better in recognizing them and then solving them. And one more click to what the actual challenge in the end for us still is. It's transforming into a value creating product company that is really future-proof given the situation that we have in the automotive market. So we are on our way. We make good progress in getting there. And now we are really interested in your questions and in your feedback and thank you for your attention.

00:28:22

Thank you.