Shifting the Paradigm From Quality Gates to Continuous Quality Assessment in Automotive

Good afternoon, everybody. My name is Rainer Dammers and I work in the automotive industry. I will introduce our company a little bit at the beginning, and I will be talking about a paradigm shift that is currently underway in the automotive industry: from a very strict, gate-driven quality approach to more of a continuous conformance approach.

Quickly, a little bit of background, because ETAS probably is not widely known as a company. We had our 25th anniversary recently and are basically focusing on providing operating system middleware, as well as tools for measurement acquisition and data processing, for the software development of deeply embedded control software within the vehicle. So I am not talking about brand portals or infotainment systems in the vehicle, but really the software that drives your vehicle. This is important to understand and distinguish from other types of software.

We are represented in 12 countries, in over 30 locations, and also have an additional brand that is becoming more and more important these days, which is ESCRYPT, which provides security concepts and malicious-activity detection and vulnerability detection. We are currently around 1,500 employees. As you currently see this in other spaces of the automotive industry as well, software competence is being concentrated and bundled. The same is true in the Bosch corporation as well, which we are part of. We are a 100% subsidiary of Robert Bosch GmbH. We have a different brand for a purpose, because we also sell to direct Bosch competitors, to other Tier 1s. We did not want to do that under the Bosch brand. That is why we have our own brand. With that consolidation of software expertise into ETAS, we will grow on the 1st of January next year significantly again. There is continuous growth over that time.

What is the mission really? We see a massive transformation in the whole automotive industry right now, from a hardware-centric, physical steel type business to a software-driven business, or software-driven product, more and more. ETAS stands now for Empowering Tomorrow's Automotive Software. Again, the focus is not that we are building the software that then powers the vehicle itself, but we provide the non-domain-specific components as well as development tools to the OEMs and other Tier 1s like Bosch, Continental, and others to do that.

We do that mainly in the mobility space, and also some related spaces like train and rail, so anything that basically moves on street, off street, and in construction. This is a little bit our portfolio. You see a lot of these things focus on in-vehicle software. There is also vehicle cloud services as well as end-to-end integration parts, as many of these functions now, with the growing complexity, cannot be handled anymore in the vehicle itself, or there is value-add to be had with services that then go to backend systems and come back into it. But the focus of what I am talking about right now focuses on the in-vehicle system.

Where did we all start? The interesting thing is that many of the concepts we are talking about in this conference and also over the last years, like lean, agile, go to Gemba, theory of constraints, etc., all originated more or less in the automotive industry in the 50s and 60s. It is well known to that industry and well established there in a way. However, that all comes from optimizing the production process. The production process is really optimized in a Taylorism environment to repeatedly, at very low cost points and very high quality, reproduce the same identical elements 100,000 or a million times over and over again. That is where all these process improvements come into the industry.

However, it has a very clear separation between engineering process and production. All these optimizations were made mainly in production, known from the Toyota Production System, but did not necessarily apply to engineering, especially not to software engineering. You have to understand that when you produce something a million times over and you are competing with a competitor against a very detailed spec, the OEM can actually switch suppliers relatively easy. You are competing predominantly on a cost point of producing that component. So software, for the longest period of time, was not a value-creating element. It was purely a cost factor. Continuous maintenance and CI/CD pipelines were not a topic. It was basically written, somehow zipped away for an archive, then flashed onto the final device, and then the device was produced a million times over. Software was viewed only as a cost factor so far, and that is the reason for a lot of the behaviors and a lot of things that we see today.

There is a very rigid process to check the quality of any component or any element that is developed very rigorously, because of course there is a need for that. You do not want a system in your vehicle to take three or four or five seconds to react, for instance, if you hit the brake pedal. It has to be reliable 100% of the time, all the time, and in certain parameters for safety reasons. It is highly regulated with regard to safety requirements and other requirements there as well. When I talk about release, what I mean is release to production. There is the famous SOP, start of production milestone, that is really important in this industry, that everybody has to hit and meet, because that is when engineering ends and it goes into the shop floor, into the manufacturing facilities, to get produced over and over again. That is what we are dealing with today.

The other thing to understand a little bit is when we talk about software in many cases today, at least this is control software. Many professionals in the auto industry who are called software developers are not software engineers in the way we understand this necessarily. They produce control software. They are highly skilled mathematicians, mostly applying math, analyzing sensor data, a whole matrix of multidimensional sensor data, and writing algorithms that then produce actuator signals as a result. They often have a physics or a math background, but not necessarily software engineering, and then they sit in front of an editor and have a compiler tied to it in the back, and that then produces software.

Now this is all changing. We will see a paradigm shift in the middle. We see the need to update things more frequently, starting from infotainment systems, for instance. This is already happening today. When we look at typical corporate directives, for example, how should things be done, we see sentences in there like this: at the release decision point in time, at that milestone before start of production, before SOP, a multidimensional or multifunctional team comes together and goes through a long checklist that says everything is there to be able to release to production.

When I joined this company and this industry in 2018, coming from an IT background, I sat down with a quality manager and asked him how he plans to scale this approach of sitting around the table to multiple releases per day. He literally went pale at that point trying to think how I would scale that with manpower or any other ways. So we needed a different way, and this is obvious by the change that is happening in the automotive industry right now.

We have to understand that the automotive industry is producing things, and as a result in every vehicle today there are between 300 and 400 compute units. Everything from your windshield wiper to your light systems to your injection pumps has its own compute unit, and it is completely isolated from all the rest of them from a functional perspective. They are only connected by a signal bus. The sensors send sensor signals onto that bus, and then it gets calculated what the result should be in real time, and then the actuators get actuator signals on that bus back. But the software is not interacting with one another in the way we are used to in terms of function calls or maybe microservices or things of that nature today.

That approach is becoming too complex and does not scale anymore, really. In the times now of assisted driving systems and autonomous driving, this architecture has to change. We are in the current transition that effectively the entire vehicle architecture internally, that none of us really ever see, is changing from this large number of distinct functions completely isolated from one another to more of a vehicle-computer type architecture. A lot of people say that will be a smartphone on wheels. It is probably the easiest analogy people can understand, but it falls a little bit short, to be honest. Even then, in the vehicle, we need redundancy. We need the real-time requirements and safety requirements all to be fulfilled. So more likely it is not going to be a smartphone on wheels, but rather a mini data center on wheels. You still have to have redundancy systems, hot failover, other kinds of things. Now imagine a small data center on wheels with SLAs of effectively 100% uptime, no failure, and real-time requirements. That is what we are dealing with.

To address that we need to change the way we work. We cannot work in silos and isolated as we do so far. This change in architecture will also now mean that we can separate hardware from software deliveries into the vehicle, basically for the first time. Not only can we expect that to happen, it is inevitable. It will happen, and we will then on this vehicle computer, more of a central computer unit, see software of different vendors running together on one platform. That is a complete novelty to the industry. We need to learn how to work together differently.

To achieve that, we work with the parent company to put in place this Continuous X, continuous everything policy, and how we want to work together to achieve that. There are many things on there that you will recognize in terms of agile, lean, and CI/CD concepts that we have seen today. I want to focus on the attempt to always maintain a release-ready state. Basically have everything in place continuously so that you could at any time pull something off the shelf and release it if need be, and focus on that.

To do that we needed to go through all these checklist items and questions at this quality gate at release time, just before the start of production. We went through this and analyzed it. Depending on what type of software you are talking about, this can shift significantly. If it is an infotainment system, you have a lot less legal and conformity requirements, for instance, so that bucket may be a little bit smaller. If this is brake, driving, or steering functions, then that part is bigger, obviously.

Much of that stuff is already in the pipeline anyway, so currently it is annoying to the engineering teams, because besides having automated test suites and test setups in their CI/CD environments, they then have to go and fill in a checklist again manually at the end to provide the same evidence that is already there anyway. For that part we have great potential to improve the life of engineers and developers and project leaders or project managers, take hours out of their schedule, and reduce redundancy.

There are other things that are not so easy to automate per se, at least not the results or the content itself. Do we have the right prices in place? Do we have embargo countries that we cannot sell to, and has that been checked? Also some of the legal conformities, like ISO 26262, the security requirements, for instance, require specific tests, long-running tests, expensive tests to be run on physical samples, basically pre-produced vehicle models. They cannot easily be pulled into a CI/CD chain, but what we can do is look at what evidence is there that these things have been completed already.

In the end, the complaints by many engineering teams were that there are too many of these questions. It is a whole list of hundreds of questions and many of them seem redundant or not necessary. We came to the conclusion that all of them seem to have their purpose. It is not like in many cases that things just got added and nothing got deleted. That is not the result that we saw. It is fairly up to date. All these questions are necessary, but many of them, as we said, are already answered before somewhere. The question is: how and to what extent can we automate collecting them?

What we are currently doing in my team is looking at reducing the amount of time and effort spent manually, while at the same time allowing us to get to this situation where we can always maintain a release-ready state, or close to release-ready state quickly, and improve quality and save cost at the same time. Basically, try to take this check that is the end quality gate just before release, pull it back, and make it part of the delivery pipeline as any other step that is there too.

If you think a little bit further, quality gate assessments are nothing else really but tests. You define what the acceptance criteria should be, you execute against that, and then you check against today. We do this all manually. In software development we already know for a long time that doing this manually is not efficient and could be done better.

What are we doing now? This is where this really is a paradigm shift in a sense, and I am not going to talk about any tools or any technology here, because that is not really the focus, as we heard this morning several times already. This is really about how do we work together? How do we communicate? How do we get to the point that the teams can act together efficiently?

Today, we have this quality gate meeting at the end of the engineering cycle. We want to shift this left, at least from the point of awareness. Instead of having a very long development cycle, today often still in the range of months to years, and then at the end trying to collect all the evidence and being surprised sometimes that maybe things fell through the cracks and were not done at the time, somebody expected someone else to do it and take care of it, and finding, okay, we have a gap, what do we do now, we do not have time to fill that gap.

We want to bring this left and look at whatever we can distill already from the regular development and delivery pipeline, collect it together automatically, and have a known state of where we are for any item that is on this list at any given time: to know what still has to be done, which documents for instance may have expired in the meantime and have to be reviewed or checked again, have this nightly process, and then base the release decision off of that. That is what we are currently working on.

One way to do that is that we can then pull this in and do all these quality-check assessments. Basically define them as a test suite in classical test-management systems that we all use, and then have scripting or anything else be executed and run to check this evidence in the background. Therefore, such a dashboard could either be a dedicated separate dashboard or we can use a test-management platform as a dashboard too.

What did we find along the way that was the most difficult, and that I am also here to get some insight or ideas or advice on from this community? We basically found, as I said earlier, we thought maybe it was a technical problem. Today there has to be a signed price sheet. Often that is a physically signed price sheet that is then scanned in. How do you check that? Of course, you can change that and have a PDF, for example, with a digitally signed signature on it that you check against a list of authorized signers. That is okay.

But what we are struggling with the most is really the historically grown organization and their responsibilities, and also the change in the job description and responsibilities of the quality-responsible people. They are used, and have been trained now for decades in parts, to be the gatekeeper, to stand there and say, nope, you cannot release, or yes, okay, I will let you through, you can go to the club, you can release. That is their job. That is their self-identification.

We need them to rethink that and sit down at the beginning and throughout the development process with the development teams, to build in those checks into the delivery pipeline wherever possible, qualify the delivery pipeline, then do spot checks that indeed the results are as expected, and readjust whatever is implemented in that delivery pipeline. They need to become basically a quality advisor and a quality supporter of the value-stream-aligned teams, of the multifunctional teams, and be embedded in those teams.

That is currently organizationally separated, and from the mindset and understanding of their job description there is big anxiety that people do not feel prepared for that shift, maybe do not want to do that. In that sense, again, it is a human change. It is a cultural change that we are mostly dealing with, with all the resistance that we saw from CG this morning too, with people having anxieties and having stress and pushing back against such a change. That is what I am here to learn: how others are dealing with these kinds of situations and what we can apply.