Trials and Tribulations of a DevOps Transformation in a Large Company

Hello, my name is Ash Noble, a chief architect for DevSecOps at Honeywell Connected Enterprise. Today, I'll be telling you about our transformation journey to a DevOps culture.

Honeywell Connected Enterprise is a relatively new organization created to dominate the industrial IoT market, bringing together the common parts of several diverse verticals such as Connected Aerospace, Connected Industrial, Connected Building, Connected Cyber, and several other divisions. Honeywell is a large company. We have around 10,000 software developers. Of those, about 1,000 are in the Connected Enterprise.

The mission that we were given, relatively simple, is to bring DevOps to the organization. We were fortunate enough that when we started this two and a half, three years ago, a book called Accelerate was released. This really gave us a blueprint for how we could start the transformation to the DevOps culture.

There were already several approaches in play for doing Agile transformation in the company, so we focused on the continuous delivery behavior and the enablers for that, such as test automation, shift left on security, continuous integration, et cetera.

In order to map out where we needed to go, we needed to understand what the problem was. So we researched a section of the pipelines that we had, 414, and found that we had 649 combinations of tools and processes within those 414 pipelines. We had some monoliths which had the same job being taken up by several tools in the same pipeline.

We found that we had a lot of commonality, but also a lot of variance. For example, our version control tools: most teams were using Bitbucket. Three-quarters of our teams were using Bamboo. Under half of our teams were using our standard deploy tool, which was Octopus Deploy. We also found that even though some teams were using the approved tool, they were using their own instances of it. They'd stood it up for some reason of their own.

So we understood where the problem was in terms of tools. We also wanted to understand what the teams were facing with their capabilities. So we got all of the teams to fill in a survey around their pipelines, measuring 13 different capabilities, roughly aligned to the capabilities of continuous delivery. On the left-hand side is approximately the build phases, and on the right-hand side is approximately the deployment phases and full automation of those capabilities. In the middle is around automated testing, some of the security.

So you can see that we had some areas we were doing really well. We had a lot of good automation, some great capabilities. Other areas, we weren't doing very well at all. And in some cases, we had pockets of excellence, but they weren't spread out across the products.

What we wanted to do was make sure that when someone created a new pipeline and released a product, that they were green across the board. All the measures were green straight away without having to do any extra work.

In order to achieve this, we came up with our three pillars plan in automation, enablement, and measurement. We wanted to automate all the tools and processes to remove manual steps for CI/CD pipeline creation and execution. We wanted to measure the key indicators to identify the areas of improvement for the development teams and the coaches. And we wanted to guide the establishment of a DevOps culture, aided by the automation, informed by the measurements. So these three pillars all interact with each other.

On the automation plan, we created a tool called AutoMate, an imaginative name. It was a self-service CI/CD pipeline portal.

In general, when you look at a pipeline at a high enough level, most teams' or most people's pipelines look relatively similar. Our pipeline, basically, we usually use an Atlassian stack, and we deploy to mostly Azure endpoints in the cloud. We also deploy to on-premise. We have a number of tools listed here, but these are just some of the major tools. We have hundreds of tools used amongst the pipelines for different languages.

So this was the standard pipeline that we wanted to create. In order to do that, we created the AutoMate tool. The AutoMate tool is used by a developer to create a CI/CD pipeline. That CI/CD pipeline uses a build farm that we supplied, which deploys to, in general, the application hosting environment, which for us is Kubernetes. It could also deploy to other Azure or on-prem environments as well. And we use the DevOps dashboards to visualize the CI/CD pipelines and help teams improve and understand their current performance.

The build farm and the application hosting was a real time saver for teams, because when they used to build pipelines themselves, they would have to create their own build agent that would then do the builds for them, and they'd usually only create one of those. And then they would also have to figure out how to deploy into a cloud environment by themselves. So we provided those for them, which dramatically reduced the amount of effort they needed to spend.

So we know we needed to create pipelines. When we analyzed all the different pipelines, we found that we had several categories of pipelines that were being put together to form an application. In general, we found an application was made up of a web UI, which was talking to some sort of web API, which had some sort of back-end processes, which were processing messages or events that were flowing through the system. The UI had some sort of widgets that they would share amongst themselves. We also had back-end libraries that would share code amongst the different back-end services. So this is how we would generally build an application.

So if we were creating pipelines, we wanted to be able to create a pipeline for each one of these categories or templates. So what would we consider as part of a template? Our templates come with sample code. If it's a to-do application, our templates all come with sample code, and it's always the same example in the sample code. If, for example, it's a web API, then it would be the to-do application, and there would be endpoints to hit to create a to-do item, to delete a to-do item, to mark it as checked, to edit a to-do item. We provide these examples in multiple languages and multiple different technologies, such as web API or web UI.

Continuous integration is a major capability that we wanted to enable our teams, so we enabled feature toggling as part of the templates out of the box. We also included many different types of automated testing, from unit testing, integration testing, acceptance testing, which is tests running against a deployed instance. We also had performance tests which were running against the deployed instance as well, and these would come out of the box.

Also included was monitoring, so the ability to monitor the application and receive alerts when things weren't working, but also assess and analyze the performance over time.

As well as code, we'd include all the scripts for the different languages and different operating systems that developers were using. So Windows, Linux, or Mac are all used throughout the organization. The scripts we'd provide were build scripts, packaging scripts, deployment scripts. We'd also include all of the scripts to do the security tool integration.

Alongside the scripts and built into the scripts would be our processes, our release management controls, such as making sure that it wasn't possible to merge to master without having a code review or at least having a successful build, and it wasn't possible to commit code which had a critical vulnerability. If a critical vulnerability is attempted to be merged, then the build will break and indicate that the team must fix it before they can merge the code.

So that was the definition of a template. We've been building out templates for a while now in multiple different languages for different categories and templates. We have web APIs in multiple languages, processes, libraries. We've got mobile development. We have some infrastructure and generic templates, a suite of data science templates. Recently, we've been adding system test, both performance and functional templates, and we also have a set of embedded templates.

This is an example of an embedded device being developed on the AutoMate system. This is the acceptance test where, when the build executes, it deploys the firmware onto the device and then runs a set of acceptance tests to indicate whether all of the features that have been added continue to run.

As part of our build phase, we used the Cake build system in order to standardize across all of our pipelines and make it easy for teams to move from one group to another group, not have to change different systems when they move.

We used a build farm to parallelize out all the tasks that needed to be run. A major concern of developers when we added all of the tools, such as the security tools in shift left, was the length of time that the build would take, and this was a valid concern when they often only had one build machine and things ran in series. When we have a build farm of 40, 50, 60 machines, then they can just parallelize out and be running in parallel and then come back together when they've all succeeded.

For our build script, we had a number of requirements. The main one was that it should have a minimum repository footprint, meaning that there is a bootstrapping script in the repository with a little bit of configuration, and the rest of the build scripts are downloaded at build time, which allows us to maintain, improve, update, fix bugs over time without having to go to every single repository and update them. We also had the requirement that they should all be able to run locally as well as on the build machine so that developers could test everything that was running on the build on their machine.

For deployment and operation, we supported multiple environments and, in some cases, multiple tenants within those environments, support multiple cloud technologies, public, private, on-prem deployments. We also automatically configured monitoring for particularly our Kubernetes environments so that the teams could make sure that their applications were running the way they thought they were without having to set up their own dashboards, again saving them time.

So our progress so far: each pipeline created with AutoMate saves around about six weeks of initial effort plus ongoing savings. When we added up the time that it took to create each of the steps, set up all of the processes, set each build script up, we found that that time, at least at Honeywell, took a significant amount of time to set up, and it was around about six weeks.

Then there's the ongoing savings. The teams are now using improved techniques. There's a lot more automated testing. The security tools are running every build instead of at the end. So a lot of the problems that were being found at the end of a development are now found throughout the development, leading to a much-reduced end phase.

Overall, we're deploying around about 250 to 300 new pipelines a month. It's a bit cyclical, which would be an interesting research project to understand sometime. We have around 4,700 unique developers using the tool, and before I showed that we had about 1,000 developers in the Connected Enterprise part of the organization, which means that we're expanding out well beyond Connected, and managing to bring in teams throughout the organization, which was our ultimate goal.

Some of the hurdles that we've faced through automation: we had some tools that don't support continuous integration. If we take test-case management, for example, in large aerospace applications, there is often a requirement to say that certain tests have been run and passed against certain requirements. Often those tools are very manual in nature and they're not very easy to hook into a continuous integration pipeline.

We also faced manual processes before release. This is again in the large company context, where we often have operations teams, and they're used to knowing that there's a release coming in a certain number of weeks, and they have sign-offs and preparation that they're doing. This is an exercise in trust and building up the capabilities and showing to the operations teams that we can release faster, and we can release with less risk each time. And so this is a trust that we're building up over time, and we're managing to make inroads and develop the relationship to be able to release our products faster, but it is still something that we're working towards.

We also, perhaps naively, expected to have more of a contribution model from teams that were using the AutoMate pipelines. Like many open source applications, it turns out that only some developers, external to our team, really wanted to contribute to AutoMate or had the drive to contribute and improve, and other teams just wanted to use the output and then let us know when something wasn't working and ask us to fix it or when there were new features to add. So that's the automation section.

We also have our measurement pillar. Again, we were lucky because of the Accelerate book, that there was really a clear set of measurements that we could use. At a high level, we had stability and throughput measures. For our throughput, we have the deployment frequency and the delivery lead time. For stability, we have our change failure percentage and mean time to restore.

This was part of that exercise where we try and prove that we can deliver safely and quickly, and safe doesn't have to be slow.

As part of our measurements, we used a set of some Python scripting and Influx database and Grafana to display the results. Here's an example of one piece of our pipeline dealing with our deployments and the measures around our deployments. We can see that we have a number of prod deployments per week. We have some production lead time, so how long does it take for a commit to make it into production? And many of our teams have, at the moment, monthly or longer batched releases. So whilst the teams are deploying internally to lower environments, the production releases are still lengthy, but they're coming down over time.

We also have a set of dashboards around the performance. So again, over time, we can see how the applications are tracking. Every time there's a build and a deploy, then a set of performance tests are enabled and run, and we can see what the performance was for specific tasks over a test run, and then what the averages across all of those test runs were, and where it's trending.

Some of the hurdles that we faced with measurement was that some of our measures were harder to collect than we expected, and we're still working towards it. Particularly the change failure rate and mean time to restore, so some of our stability measures.

We brought a diverse group of verticals together, and those verticals were using a varied set of tools to measure their defects when they were getting raised by a customer. Some were using ServiceNow, some were using Jira, some were using Jira Service Desk, and it went on. There was up to eight different tools that were being used. And we're gradually rationalizing those, but we still haven't got to a consistent measure across all the teams, but we're making good progress there.

As part of that, we found that we could use proxy measures to measure those change failure rate and mean time to restore. We used the deployment failures as a proxy for change failure rate. It's not ideal because if there's a defect which didn't stop the promotion and deployment, then that wouldn't be included. But it did give us a general guide to how often things were failing and how often we could repair them.

We found that visualizing structured data was difficult in the tools, so we were using Influx, which is a time series database, but sometimes we wanted to include information such as the team or the organization that was building a feature and was working on a pipeline. Some of that structured data, not time series data, was difficult to include in the measurement dashboards.

Okay, so now for enablement. Enablement is really where we drive the cultural change in the organization. We wanted to have a small team of coaches. We wanted to have Spotify-like guilds, which were guided by those coaches and taking members from each team and translating information or transferring information out to those teams, and also the teams bringing it back up through the coaches, and we figure out where we need to improve and how we can improve across all of the teams.

We leveraged our existing communities of practice. Often we found that there were teams doing individual lunchtime sessions, brown bag sessions with their individual teams, but they weren't necessarily advertising that outside of their organization. So someone that was talking about best practices in unit testing wouldn't know that there was another team that had also just given something on the best way to deploy to a particular type of infrastructure. So we developed some mechanisms to allow teams to share that more widely across the organization.

We also wanted to utilize our measurements to improve a team's performance. Some of these items we hit, and some of them we're still working on.

Part of the enablement was training. We have monthly jump-starts. It took us a while to get to the understanding that we needed to run these very frequently. We thought we could just run them occasionally, get a whole bunch of developers, and teach them, but it didn't work out that way. We needed to do smaller, more hands-on, focused sessions with the developers so that they really got a chance to dig into the technologies that were involved.

We have weekly open hours, so sort of drop-in sessions. If someone's got a question about, can AutoMate do this, or how would I achieve that, then we've got a spot where they can just drop into a virtual room and ask some questions. And we also do quarterly product deep dives. Before COVID, this used to be in-person workshops where we would all come together into a particular team and then work with that team, usually legacy code, to figure out the best way to sort of modernize their pipelines into the AutoMate system.

Another part of our enablement was online support. We neglected this. We were a bit naive at the start, where we didn't include this in our original planning, and that cost us a little bit. It was very organic. It grew over time. It turned out that our AutoMate developers became the front line for everybody's problems. So AutoMate was meant to be a self-service portal. The people create their pipelines as if they were using all of our individual tools to do that themselves, and then they'd have their pipeline that they would work with.

But because they ended up with a pipeline that was created for them, we almost created a problem by making it too easy for them. We created our own problem. Teams would type in a name, choose a template type, press return, 10 or 15 minutes later, they'd have an API that was stood up in Kubernetes with an endpoint that they could hit, having run all the tests and deployed the application.

Now, there's a lot of stuff that goes on in there, and we found that teams weren't learning or didn't understand the technology. So some teams weren't familiar with Kubernetes, with containers even. But their code was being delivered and running in a container, and then there was this gap where if they needed to make a change or debug something that was going on, they didn't quite understand it yet. And so that was a change where we had to run some more training. But also, we became the first point where if there was any question at all, if the build broke, the first thing people would do is ask on the online support for some help.

Unfortunately, people didn't always think carefully about what they were doing, the questions they'd ask first, or even necessarily look at the logs. Here was an example of one question that we received. This is the full question: "Can someone help me here? There are no changes to the repo." And I was very impressed with the person that responded to this. I thought they were much more rational than I would've been. They said, "Can you specify what you need help with?" I may not have been quite as polite.

So, in order to try and deal with the fact that we had a lot of people asking us frontline support about just the technology, about different types of tools that were running in their application, we created a little bot. We called it the Template Support Bot, and that allowed people to basically use a tree of questions in order to solve their problems. So in this case, the person has drilled down into their build is broken, and then they can choose from the various areas where the build might be broken. So if it was a security job, then perhaps they've got a vulnerability that they need to address. And if they clicked on that, then they would get a list of options about, okay, here's how you understand what to do when you get a vulnerability. So that was one area where we got some good success in trying to ease the burden for us on our enablement.

I talked at the start about the goal of having coaches and to build up Spotify guilds. That didn't really pan out for us. We found that it was extraordinarily difficult to hire coaches. We at least struggled with this because often we were getting people who might've been excellent coaches but had either no or a very shallow understanding of DevOps. And when you're trying to explain to a team why they should be using continuous integration instead of feature branches, it often helps to have a good understanding, to have been there, to have been an application developer and made that realization yourself. So we struggled to hire coaches.

We also had organizational inertia, and this is something that we were expecting, but is still hard to work through, where teams have been doing things for a long time the way they've been doing it. Business processes have grown up around creating batched releases over large periods of time. Those sorts of things take time to work their way through, and we're still working through that.

We had pockets of excellence in the organization where at least one part of the organization was doing each of the capabilities in an excellent way. However, getting those teams to teach everybody else how to do it was something that was a challenge as well. We were expecting people would love to share about how they did something the best way. However, it turns out that some people didn't think that that was their job, which it probably wasn't. And they weren't that keen to spend time helping everybody, which is understandable when they've got other features to release, but we weren't really expecting that.

So overall, we provide a lot of capabilities to teams. However, yet to realize the full potential of it across all of the teams. We have a very successful automation rollout with high adoption rate. We have a reasonably successful measurement rollout. However, our enablement still has a long way to go, and that enablement will enable the measurement to be more successful as well.

So that's our three pillars. That's what we learned so far in two and a half years of bringing a DevOps culture to Honeywell Connected Enterprise. Thank you.