SRE From Scratch: An Enterprise Journey

Hi, everyone.

We are Avinash Rao and Monika Gupta, and it's a pleasure to be here at the DevOps Enterprise Summit.

Today, we are going to take you through our journey of SRE from scratch, an enterprise journey.

Hey, everyone. I'm Monika Gupta. I am SRE lead for Mobius platform and tools at BT Technology.

I have more than 18 years of IT experience in software development and management. I have been practicing Agile and DevOps for more than a decade now. I am also a certified SRE Foundation professional. From the last one year, I have been playing the role of SRE lead for Mobius platform and tools in BT Technology.

I'm Avinash. I'm VP of Products at Digite. In the period that Monika and I collaborated on this particular journey, I was part of Wipro Digital, and together we were working on the Mobius program.

I'm a DevOps Institute ambassador and a DASA certified DevOps coach. I've spent about a decade now working on Agile, Lean, DevOps, as well as Kanban.

And that's me, by the way, on my way to Everest Base Camp. Someone just reminded me that it's been exactly two years to the date that we started that particular journey.

Let's start with an overall view of what BT was trying to achieve.

What you see on the screen is the leadership ambition that BT has: that by 2030, BT wants to become the world's most trusted connector of people, devices, and machines. And there are really two or three key things that must come together for BT to achieve what it wants to.

The first one is the business strategy. And a key part of building a strong foundation of that business strategy is to be able to bring a very strong technology capability to British Telecom.

The Mobius platform was started in order to provide some of the DevOps capability to the British Telecom environment.

Overall, if you look at the journey from an end-to-end perspective, the products that BT offers are underpinned fundamentally, and this is true of any organization in this digital world, by a very strong IT.

We also must upgrade the processes by which this particular value is delivered, and together, create a better experience, both for the people utilizing these technologies inside BT and for BT's customers.

A second key item that really underpins this particular strategy is a culture where people can be their best. And what BT wants to do is create an environment where people have access to cutting-edge work and the latest technology.

And this is not technology for technology's sake, but really a way for BT to give a world-class IT experience to its end customers.

So when we started working together at the start of the Mobius program, there were a few things that we were aware of in terms of what is needed for success of what we call the Mobius program.

Obviously, Mobius wanted to create a world-class DevOps and CI/CD environment.

A lot of the environments at that point in BT were on-premise, and so we wanted to start off the cloud adoption journey.

Also, if you look at many enterprise journeys, there is a focus on DevOps and CI/CD, but not as much on continuous testing, which becomes a key problem. And hence, we thought of continuous testing as being an integral part of the Mobius journey.

Achieving a high level of DevOps capability automatically implies the need for automation.

When we looked at the overall BT strategy, we also saw that culture is an important part of this change, and that's where the Agile processes start coming in.

However, there were some challenges that we faced right off the bat. One was the lack of a measurement culture. The second was a lack of tooling governance and localized optimization. We had several teams, each with their own DevOps pipelines at completely varying degrees of capability, some fairly advanced and some very rudimentary.

Our first step was to put together a collection of open source tools and provide those as part of the Mobius platform at BT.

Yeah. So keeping all those goals in mind, we started building a framework to create a CI/CD pipeline using whatever tools we had available. So mainly those were the open source tools like GitLab, Jenkins, SonarQube, Nexus.

And we also defined the standard gates for quality and deployment for all our engineering teams to adhere to.

So in order to provide a centralized, robust, scalable platform, our goal was to move from open source to enterprise tools so that we could scale to support 2,000-plus applications.

So slowly, we started moving to enterprise-grade tools. So now we are using GitLab Enterprise version to give us a strong foundation for a source control system. Then we moved to an enterprise-grade solution for our CI solution. So we moved to cloud-based Jenkins. And for our deployments, we moved to Ansible Tower.

So considering all this, we thought we had built a very robust platform for our engineering teams to use. But I'm sure with all this, you might be wondering, we might be having a great platform that can be used by our BT engineering teams. But no. We started seeing a lot of complaints from our end users who were using the platform.

So they were saying the platform is not stable, masters are frequently going down, a lot of 500 errors are coming. So this was the time when we felt there's a need for us to retrospect.

So originally, how we were working, there were four teams. We were working as four different teams. So there was an onboarding enablement team who was helping our end users to set up the pipeline onto our platform. Then there was a platform team who was responsible for creating a standard pipeline with the standard stages, gates, et cetera. And then there was a core tools team whose main purpose was to provide good features related to each and every tool. And there was a lean support team on a daily basis to support customers for providing them access or setting up webhooks, et cetera, for them to run the normal pipeline.

So we realized all of these four teams were really not working in collaboration. So there were a lot of silos.

So then we all leads got together in forums, and we bought SRE books to understand SRE principles. We got ourselves certified. We also hired one SRE consultant to help us assess our current SRE maturity state. And also we wanted him to help us with that adoption path.

So we followed this four-step SRE adoption model. We assessed our current state, defined our to-be state. In order to bootstrap SRE culture in our team, we set up some workshops, trainings, et cetera, for our team to actually understand what SLOs mean, what availability means, what exactly service reliability means. So now we are more in an adopt phase where we are trying to scale to adopt these SRE practices across all our services.

Building on a platform model that was available from a partner, we looked at what we need in order to really achieve the performance metrics that were important to our customers.

So there are two parts to that. One was with regard to speed, and the second with regard to stability and reliability. The key metrics we picked up from a speed perspective was lead time and release cadence.

And from a stability perspective, it was the time to detect, engage, and restore from an incident. Because what used to happen is that the Mobius team realized that there was a problem when an end customer reported the problem. So it was really important for us to be able to proactively detect and then engage and restore the issue or the error. And of course, we had to keep a really close watch on the change fail rate.

Now, underpinning this entire change was to bring in a culture that was focused more on Agile and Lean values. As Monika just spoke about the structure, we had created multiple silos, because of which there was no end-to-end value stream thinking, and hence that had become a major impediment to achieving flow across the platform.

In terms of our movement to the cloud, cloud clearly offers several significant advantages in order for us to be able to provide additional requirements in terms of auto-scaling when there is a higher demand. And so it was an important component of what we looked at.

From a reliability engineering perspective, we first had to take a very close look at where we were that day in terms of service level management, in terms of monitoring and observability, toil management.

So given all these, we obviously couldn't do everything at the same time. So we took a good look at each of these, prioritized them, and created a subset that we should start with.

Yeah. So after assessing our current state, we decided to start slow, but do the right thing that was making sense at that point in time.

So we focused on these four key areas, like service level managements, where actually we wanted to measure in terms of availability where we were, and we wanted to set a very basic monitoring. So we were very reactive in nature when we started, and we had no system in place that could let us know about the problems well ahead. That's when we wanted to set up basic proactive monitoring.

And the other thing we wanted to focus is we wanted to improve on our incident response model in order to address customer issues and queries on time. So we wanted to provide them updates, right updates on right time. So be it related to incidents or be it related to any change or upgrade that was coming up.

And fourth area, we wanted to focus on the toil.

Keeping this in mind, we set the goal to move from reactive to proactive and be more preventive in future. So let me now take you through our reactive to proactive journey, what we wanted to achieve in six months.

So currently where we are: we have set up a good monitoring in Dynatrace. We have moved to a single ITSM platform, ServiceNow, where we are able to manage all our tickets, like your incidents, change, problem, everything. We have implemented a clear comm strategy to send enough updates to our users on time. We are able to measure some of the key metrics like availability, throughput, mean time to detect, mean time to resolve.

We have set up a good knowledge base for our users and for our L1, L2 engineers to follow. So as a result, we are seeing good amount of problems identified by Dynatrace and being reported as incidents and improvements in ServiceNow.

So now we are able to take right decision to address these problems in terms of raising CRs, problem tickets, et cetera, with our SMEs or with our infrastructure specialist.

So I'm aware of the improvements that happened as a result of some of these things that we did together, Monika. But given that I've moved on to a different role now, it'll be interesting for me to understand what really are the next steps that you are looking at in order to continuously improve the platform.

Yeah, Avinash, so you're right. So we are still learning and improving. There are a lot of areas that we still need to improve.

So until now, we were working as a centralized SRE team. So as a whole, they were focusing on the service reliability issue for all the services. So we are seeing that model isn't working very well.

So what we are planning to do is to set up a centralized, federated, reliability-conscious team wherein we will align SRE engineers with our key services, so as and when there is any new feature or upgrade planned, these engineers will make sure those features are robust enough. So they will be driving the non-functional requirements. That is one thing that we would be bringing in.

And next step, we want to take our monitoring to next levels. We want to bring in good observability by introducing good log monitoring using ELK Stack, and also we want to introduce AIOps.

So with all these issues, et cetera, flowing in, we want to now analyze the trend and make use of those trends to give a good capacity prediction. So say for six months or one year down the line, how much our services in terms of infrastructure it has to scale. So we will be looking at that data.

And also, we are very much on-prem heavy currently. We want to move to cloud. So we are really trying to analyze in terms of what SRE practices we need to bring in order to support our tools on cloud.

So these are the key things that we would be focusing on in the coming next three to six months.

One of the things that I've learned as being part of this journey along with you and BT is that in the beginning, we did look to see if there was one SRE tool which we could bring in and make this entire process happen.

But I think the real learning which we would like to share with our audience today is that you do have to see what is necessary for your unique ecosystem and to be able to do the specific interventions that are needed to give the improvements that your particular situation demands.

With that, we are signing off, and we are happy to take any questions.