The DevOps Journey in an Enterprise

Thank you, Gene, for having me at the DevOps Enterprise Summit. This conference, along with many of your reports and books, has given me and my colleagues so much inspiration. So when you asked me to join as a speaker, I of course just couldn't say no.

Hi, I'm Anders Lundsgård from tretton37, the Stockholm office. I am a cloud solutions architect with a real passion for DevOps, and I'm here to share my story about the DevOps journey in an enterprise. That enterprise is Scania, an engine, bus, and heavyweight truck manufacturer.

My passion in my working life is to enable developers to work much more efficiently than I have ever been able to myself. I teach developers how to operate their code in cloud environments, and I also try -- which has shown to be a little bit more complicated -- to teach operations people how to code.

Since I've been involved in the cloud adoption at Scania for six to seven years, I see cloud as an enabler, a technical enabler for DevOps.

A disclaimer before I continue: this is not a way for me to define DevOps. I will share the story that me and hundreds of other colleagues at Scania Connected Services have made during a 10-year journey to work more efficiently with how we produce and put software in the hands of the end users.

Then we have had DevOps as a buzzword to Google about, to have some opinions about, and I think it's really good. We have these good books like The Phoenix Project and The DevOps Handbook, but we do not have a single page like, for example, the Agile Manifesto. But that's fine. I think it's good that everyone can have their own opinions, and I guess that's the reason we're having these kinds of conferences.

If we step back in time, my very first job was at a startup. I was the only guy at a company of two that knew how to write some code. So I wrote the code. I wrote the tests. I made some kind of deployment pipeline to push out the code into the web server, and I also put on some key metrics so that I could see that my end users could get some web traffic, at least. I was the guy that the end user called when something went wrong.

DevOps was not coined back then, but it was a kind of DevOps situation because I worked with the code, and I ensured that the end user could use that code.

In 2008, I decided to move to an enterprise, Scania, and I was faced with a totally new situation for me as a developer. When I started the first week, it was explained to me that the code I was writing right now was not going to be deployed into production until the next year. Nine months after I wrote the code, that would go live into production. I thought that maybe this is how it works in large enterprises. But during the years, I've come to understand that it does not have to be that situation.

I also understand that this tug of war we see on the screen is between developers with one kind of key metric and operations with another one. It is not a technical challenge. Of course there are technical challenges, but most of the challenges have absolutely been cultural, and I will try to focus on those in this talk.

Here's a picture taken from a very big moment. It was the very last manual deployment to production back in 2015. It was me and about 10 other engineers that went into the office on a Saturday morning to do the release.

This was a very, I would say, non-DevOps situation for many reasons. One, we can see at the left, we had a release plan. A release plan stated to us what we should do to have a successful release that Saturday morning. At the top of the release plan, it stated that we should call the network guy to bring out the servers from the load balancer and turn on the maintenance page, because it was the fact that we had downtime for the end user while we were doing the release. That was the reason we went in early on a Saturday morning.

Then on the screens, we have four or actually six web servers that me and the other colleagues moved some files to, to put live in production with a new version of the code. Behind me, we had the DBA that at some point in the release plan got the responsibility to do the schema changes in the database.

In the middle, we have a small Post-it, and on this Post-it it was the phone number of the engineering manager, because when things went wrong -- which sadly happens sometimes -- we needed to escalate it and call in some developers that could help us troubleshoot and find the issue.

The Red Bulls to the right might just be a symbol that this was a big event. We needed to go in on a Saturday morning, and we did it only once a month, and it was a stressful situation.

So we had this setup that probably many have seen: a wall between the development and operations department. And it was actually not one office space at Scania. I was in one building; for operations guys, I had to go 20 minutes to another department to meet these guys.

We from development needed to have more features out more quickly, and operations needed to keep up the stability. These two goals have shown to be very often in conflict with each other.

The guy at the top of the wall here was me, sometimes actually finger-pointing to the operations guys because they were the bottleneck in our release improvement process.

Today, I have turned out to be that security guy. Since I've moved into cloud adoption, I've also worked a lot with security configurations. And I actually have that developer still on my shoulder telling me that we really need to increase the pace, reduce the batch size for each release to production. But on the other hand, I've also seen that if you only let loose all configurations in a cloud infrastructure, bad things can happen.

If we step back to the date when we had manual deployments, we formed a vision in our department: a vision that an engineer should one day be able to wake up with an idea of a change, big or small, maybe just changing the color of a button. He or she could write some tests and make the code, and after lunch deploy that code into production totally on his or her own. Maybe the next day, or hopefully the same day, he or she could evaluate if that deployment turned out to be successful for the end user or not.

What we didn't know back in 2015 was that this was actually going to happen already next year.

The key foundation, I would say, the technical enabler for us to move in the right direction to a more agile way of working, was the practices around continuous integration.

Continuous integration has a lot of pillars, but the four key pillars to us were, firstly, version control everything: code, tests, configurations, schema changes in the database should also be in version control. So the DBA had to start using Git. Later, when we moved into cloud, we also saw that we could version control the infrastructure.

Automation, of course. We should have automation, and I think everyone agrees on that. We used the release plan as a backlog, basically, to automate our deployment process. The first step was to connect to the network architect and ask, how can we automate the process of, instead of calling the network guy, having that as a Bash script or whatever? Luckily, some months later, we got an API that could be introduced into our deployment tool.

Trunk-based development -- or today, I think we should mention main-based development. Probably the practice that many of us have missed is that each developer on a code base should deploy or merge the code into mainline at least once a day. That means no long-lived feature branches.

Claim over blame. Now to this picture with the lamps in the ceiling. Green obviously is a good color. Yellow is actually when things have been a failure. We had a failure, but it's actually a good color because something went wrong, the red light went off, and someone took responsibility to fix that issue. So there's only one bad color here, and that's the red. If the red lamp is red over 30 seconds, then we have some problem. We have a failure that no one cares about.

This way of highlighting this was very good because then we also got management into the daily work. They can see that we have issues. Probably it was a very stressful situation for the development teams that ended up having a red light for a long time. So reducing that time when the red light was on was a key metric for us.

I have to step back. Developers: version control everything except secrets, okay? Version-controlled secrets can of course mean secrets come into the wrong hands, but it also reveals that you probably have some things to do when it comes to automation.

Some metrics in our deploy frequency at Scania Connected Services. Back in 2011, we had software projects, two and three in parallel, and about three deployments to production every year. The very last manual deployment that I talked about was in 2015 when we had agile teams, one deployment per month. The key enabler for that step was continuous integration.

But then something happened in 2016. This deployment metric went high up through the roof: 30 deploys per day. The key enabler, I want to say for this, was a microservice architecture where each development team owned their own part of the system and they could independently deploy those into production.

We challenged and improved infrastructure-related processes like, for example, the network configuration. But we also cannot have a change management meeting every second Tuesday if we want to do 30 deploys per day.

Trust and courage from management: previously, we had go/no-go live meetings before the Saturday morning, stating that our senior manager needed to approve the deployment to production. Those meetings are gone, and probably they are gone forever.

I must say, before I continue, this was all on-prem. This was not the cloud that enabled us to do these kinds of deployments this frequently.

Short about the branching strategies and the evolution over this time. Back in 2011, we had mainline, release branches where we did the release, and a bunch of feature branches. The evolution was that we could remove the feature branches, still had a code stop before a release, but the true improvement was when we also could remove the release branches.

We talked a lot about autonomous teams. We should have an architecture that enabled teams to have their own code base, their own part of the system, and those parts could be independently deployable compared to the other teams. Of course, that's a beauty thing because the teams can choose the tooling, their frameworks, and the languages for their particular challenges.

But for me, as a release engineer, the best thing was that if something goes wrong with one team, the other 29 teams can continue as usual. Of course, we should take care of that team once things went wrong. But in the past, this failure two days before a Saturday morning would mean that we had to block the release and reschedule, and everyone was affected.

Zero downtime was one key technical enablement that we did in 2016. That means basically that we do a production deployment and the end user can continue to use the system. That sounds good, but the best thing with this, I must say, is that this enables us to do deployments on a daily basis. The engineers that write the code will be present and actually will be the ones responsible for pushing the code into production. So when things went wrong, engineers can be there and solve the problem directly.

Immutable production is a very interesting topic. I will not go deep into this, of course, in this short time. But when I say that developers should not log in to production servers, operations guys tend to like this. But when I continue and say that you, operations guy, should not log into those servers either, then this discussion turns into another area. With immutable production, we need to have the practice of version-controlling the infrastructure. If we should roll out a new configuration change into a server, we change that configuration in version control, and then we ramp up a totally new server and kill the old one.

Remove handovers. We should really avoid sub-optimizing. I think our release plan was one thing that showed us that we had many sub-optimizations, because the day when we started to collaborate and show the release plan to the operations guys, then we could start to make real improvements. So try to avoid sub-optimizing in organizational silos.

We realized that we could decouple deploy from release. Deploy, to us, means that we move binaries to the web server, to the production servers, or we can move the JavaScript file into an S3 bucket that is the front end for our application. Release means that we enable some new features, or perhaps a totally new service, for the end users. Deploy is 100% a development team concern that they take care of. A release might be synchronized with some marketing activities, or documentation for the end user has to be written. So that is a business decision.

This was late 2016, and this was a big release. We enabled a totally brand-new Angular front end with a microservice backend. This product owner enabled this site and system for 10 markets. So we made a graceful rollout of this system for a period of two or three weeks. We enabled this new front end, and when things went wrong -- luckily it didn't -- but if we had some struggle, it would be very easy to do a rollback: just uncheck the market and the old site would still be present.

So we ended up going into the office on a Saturday morning, but there was one time we had a special demand from the UK market. For some reason, they needed a deployment or a release on a Saturday morning, the 1st of September, 2016. This little girl, Tilda, she was four by that time, got the responsibility to do a release. Of course, she didn't really know what she was doing. Click a button. She didn't think it was that fun.

But you can imagine me standing behind her, having my four-year-old daughter doing a release, when I, six months earlier, had gone into the office together with 10 other engineers basically achieving the same thing with hours of work. I was, of course, super excited about this.

Affinity is something within the culture of DevOps that is extremely important. For us, that meant that developers need to understand the complexity of operating software of all kinds, especially in production environments when you need 24/7 and have SLAs up to 99.99%. There are such things as availability. You might, as a developer, deploy your code into at least two servers if we have one power outage. There are things like durability, scalability, and security that operations people in a traditional on-prem system take care of.

On the other hand, the operations person needed to understand the complexity of dealing with different code states. What I mean with that is that we have developers; they produce code that is live in production. They continuously have new demands that will probably enable new code to be written. Hopefully, we have deleted code that didn't make any good for the end user.

But the hardest state of code is code that is pending. That means code has been written, checked into version control, but not yet deployed into production. The longer the time it takes for the developer to check code into version control until it goes live, the bigger the bang will be when the code goes live and eventually doesn't work. It is also the case that if we have a long time, probably many developers will push code at the same time, making it even harder to troubleshoot. That is something that has shown to be very important for operations people to find affinity for developers.

Scania got a cloud-first decision back in 2016, and that's a big change. Of course, we got a state-of-the-art infrastructure platform to work with, but it was a very big challenge for the organization that had an on-prem way of working.

Imagine we have called the network guy on a release to do some change in the load balancer. Now, suddenly, that load balancer, from being hardware and some clicks in a web UI, is suddenly something that is defined in code in a way that the developer is used to seeing code, and that code can be in version control. So this is a big shift.

Our network, backup, and DBAs need to understand that we need to define infrastructure, help the developers to define it, version-control it, to make it reliable. I worked at the cloud security team for two years, and I've been working with helping developers to define their own infrastructure in the cloud.

Here are our core pillars for how we have been working as a cloud security team, a team of three people for over 1,000 developers. First of all, within our cloud-native strategy, we also have a cloud security strategy: we should use the cloud-native services. We as a security team, even though we are in a large enterprise, cannot build better and more secure services than the cloud providers. Cloud providers are experts in security.

We should share failures, of course, within the organization but also externally. Ubiquitous encryption: if we can encrypt things, we should do it. In a cloud environment, we can very often encrypt it in one single line of code. We also had some happenings around security: find security champions, have a day when developers can compete around security.

But the biggest two parts that have been really important to us have been: we as security professionals really need to get out of the cave. We cannot only say no. We can say no. We at the cloud security team had a compliance framework stating that you should, for example, not have port 22 open for the whole world. But we put on ourselves the responsibility that if we say no, we should also provide a remediation for developers.

This was inspired by a true DevOps moment for me. Back in 2015, I started to collaborate with our network architect, and we saw that we needed to define the infrastructure or the networking in our cloud environment because there were some common rules that needed to be in place for the whole cloud setup.

We saw that we could define this virtual private cloud in a way that the network architect was very comfortable with. He knew about CIDRs, BGP, and so forth. And me, as a developer, I knew how to write code, and I knew that version-controlling that code in Git was a very good practice to enable automation. So we made this together, and still today this work is being used every day to create new virtual private clouds in our cloud environment.

Time is running out for me. Here's how you can help. Of course, continue to keep distance. Hopefully this year, you can take on this challenge. I challenge you to take the DevOps hug. I did this hug seven years ago just to see if I survived. The outcome was more than that I'm apparently still alive: it was also that I started to find true friends among the operations department.

I will end by saying that you as a developer, if you have any concern about where you can start, a small developer in a big enterprise, try to enable zero downtime deployments. That's my call to you. And you, operations person, if you don't yet version-control your scripts in Git, start to learn Git. The day a developer comes to you and asks for a change, and you can ask him or her to do a pull request in your Git repository for that change, you have started to enable a true DevOps moment.

That was all I had. Of course, you can find me on Slack. You can always find me via DevOps.vision and connect with me on LinkedIn, for example. So I will again thank you for being here. Have a great day and take care. Bye.