Europe 2022
We Connect for Good
BT's agile journey.
JM
James Moverley
Senior Designer, BT
Transcript
Hi there. My name is James moley. And thank you for coming to watch my session, which is gonna be about BT and our agile journey and the road we've taken to introduce continual delivery in the mobile part of the business. Now, I wanna talk a little bit about BT. So what is BT BT slogan is we connect for good and they provide up to they. They provide television services over 2 million subscribers. 14 million homes are connected by broadband and fixed line 27.5 million mobile subscribers, uh, mobile phone users, 1.2 million enterprise customers, and over 5.8 million public wifi hotspots, connecting people all over the UK. We have global enterprise customers in over 180 countries and we provide some of the UK's critical national infrastructure rolling out our next generation of emergency services network, which will support all the UK's blue light services. So prologue I've decided to structure my presentation and the sort of realms of a very familiar book that you should all know.
See if you can guess what it is. Let's talk about the status quo in every company I've worked in to date. Uh, most of them, both operators and vendors alike have always delivered in a proven model. That is waterfall. We work on projects. We use fundings that fund the projects to deliver change to the network where identified necessary. There's a program board that identify which projects take priority and where the capital should go. Usually that that sort of wheel know leads to limited resources, a strain on people and cognitive lows between teams. We also seen the long wait times as we hand over work between departments. There's a test department, there's a design department, there's an operations department. It's classic territories until recent years. I didn't realize that there was a problem. This is where you refer to it was business as usual chapter one, this point in my career, I was an automation engineer.
I had taken on board operating unit systems and virtual machine environments. I'd been looking at N E in the deployment of systems and vendor driven solutions into our network. A lot of the stuff was kind of manually done. There were some automation here for standing up bits and pieces, and we were trying to push forward full automation, self-service portals, this kind of stuff. And in doing so, I was looking across the business in all the different areas to see where, who else had been adopting similar tooling and mindset. And it was quite interesting because a few names had popped up and I could see that some departments maybe are two years ahead of us and some even maybe a year behind us and actually by reaching out and contacting these individuals, I was able to establish several working interest groups, which basically pulled people from the different parts of the organization. The biggest one that we established was announcable group Ansible seemed to be quite well adapted, uh, adopted, sorry, and also Python. You know, a lot of people were bringing in Python to create their own tool sets. This small band of Mary followers decided to get together every Friday. And in those sessions, we would hammer out some of the businesses sort of underlying issues and challenges. And it was great just to meet really smart folk who were really key on solving business problems with their code.
Hearing. Other stories inspired me to have a look, to see what could be done, what platforms could be provided. And it was good to see that there was a need for improvement across the board.
Monday, the 30th November was the time an email was sent out to my director. It was the announcement of a new position to be filled of director of DevOps. Filling that position was an individual whose name had cropped up across the organization. And I was very keen to see him start so keen. In fact that I actually sent an email. I don't usually contact my senior management. They're busy doing senior management stuff, but in this instance I couldn't resist. I felt compelled. I had to announce the fact, this is great. This was the thing I was looking for. We've been trying to get various platforms and, and services introduced, but to no avail, we didn't have the traction that we needed. And all of a sudden we were gonna get that traction. This was our moment of change With the next two weeks. Uh, I'd spoken to the individual and they had asked me to join them on the journey. Little did I know the adventure that I was about to endure? We formed a new cross-functional squad, never heard of squad before I was told to unlearn and get used to learning. The good news was that was already something I really enjoyed. So I look forward to that. Most importantly, he introduced me to the Phoenix project. See this book behind me. I always keep it the hand. It's good to review this book. And if you haven't read it, you really need to read it.
And we were eased into the agile and scrum ways of working a scrum master was appointed and agile work sessions were put together where we get to see how things can work and how we should be aiming to work. I felt as a 47 year old, very well indoctrinated and institutionalized in the way of water form projects. And all of a sudden things started to click into pace. I actually found my Eric and my Eric, every time we spoke was making more sense than a lot of things had done for many years. So I was compelled to follow this journey
Friday, the 15th of January, 2021, we just completed our first ever sprint. Two weeks. We had a massive sprint planning session that lasted nearly two days and the squad were keen. We had all our new starters, a fantastic group of earlier doctors, people that really wanted to see the success. And we eager for change. We established a cab collaboration tooling on the various platforms, wikis ticketing systems, messaging boards, where we would chat, where we do our white boarding. We managed to expose get exposure to other agile areas of the business. See how they operated, get an idea, be inspired. We're also given the luxury of our own equipment. We stood up our own sand environment, uh, an area which we could safely experiment without causing too much grief for the rest of the business. Uh, and there's little O to the cow there because we always needed more cowbell. The journey is well and truly begun
From that point on voice February through to may, we spent the next couple of months inviting others to read with us. Uh, we had lots of internal sharing and discussions of the materials that were presented to us. Uh, we managed to generate a really good ground swell. There were various other areas and other squads being spun up at the same time, we were encouraged to be inspired. We were led to read lots of fantastic documentation and showing lots of interesting projects. Most importantly, we were given a psychological safety within our own team. We made sure that everyone felt welcome. We made sure that everyone could speak their own mind, and we made sure it did not matter about failure or success or any of that stuff. We just wanted to learn. We wanted to celebrate what we were doing. Rapid learning can only be achieve by doing
Influences and inspirations. Most definitely the very early stages. And thanks to our director who guided us on our path. The Phoenix project was where we started. I consume that book in two weeks. It's the most reading I've done end to end of a book in a long time. Then we moved on to the DevOps handbook to understand the principles, the ways of flow, the ways of learning the iterations, the unicorn project made a fantastic follow up story in the world of software delivery. Soon as say for happier again, teaching us about patterns and Antipas suddenly things were becoming very crystal clear. We could see those things popping out at us, continual delivery. I had to list this book, this book as a technical architect and as a technical developer really led me and strong the light on the actual mechanics, behind the scenes of what we were trying to achieve. Uh, Nicole fors Springs, accelerate, refines that view and gave us the sort of image of where we need to be. It really buffered up and helped crystallize the DevOps handbook for us. And last but not least, I have listed here the novel version or the graphic novel version of the goal. Uh, the goal itself is a very extensive book, but the graphic novel was something I could breeze over in the weekend to get the gist of the story. The kids loved it as well. It
Here are a few inspirations as well. I wanted to these they're poignant, the big K Kedar. He was really the seed of it all has to be acknowledged here is he holding my friend Elmo. We used to use Elmo a hell of a lot. It's enough. Let's move on. I like to acknowledge the GitLab. They also introduced us to quite a few workshops. They helped sort of get us along and introduce us to whoever developer, DevOps conferences and, uh, facilities that were around in our, in our region, UK national DevOps, fantastic conferences, uh, the better value, sooner, safer, happier community they've offered numerous great talks by various inspirational people, people. And they've also done quite a few good workshops. Let's not forget the DevOps Institute. They're re reasonably doing lots and lots and lots of skill days. There are skill updates and skill up hours, and people can dive in and see what it's ne what's needed to do what you need to do.
I happy to call out John smart, his delivery, an absolute hilarious video on certified really agile practitioner just had us in stitches and sometimes, and for some, it was a bit too hard, hard near the truth to bear. We originally proposed showing this video as a part of one of our sprint standups. It was decided against because we figured at that time it may have offended too many. Uh, I have to call out Brian Finster. This guy is absolutely the real deal. You might have heard it before. But speaking with Brian had really inspired me. And that allowed me to sort of move forward on a number of aspects of what I was delivering. This guy has really been there and he's really done it. Check out his a DMF framework and check out minimum CD. Do org define that more about that? I can also list Dave Farley, his work on continual delivery and the book and all his videos on YouTube have been an absolute inspiration, but last to not least, let's look at the DevOps by summit itself.
It was the actual turning point, I think in terms of inspiration by that summit, I got to meet a lot of fantastic people, both publishers, vendors, users, people who just wanted to share the moreover absolutely great characters. And if you were there last year, you remember us hanging out in the bar. We were chatting to everybody and hopefully we'd be there this year, too. Chapter five, the vendor swap out PT had been challenged to actually remove one of the vendors from our network for various reasons. And it was an opportunity to move from bare metal deployment to actual cloud native. The area that was being replaced was PAC core. I'll describe a little bit more about that in a second. Also a part of that mix, we're evolving new standards from the European telecom standards Institute, Etsy on the network virtualization functions and something called CSUN, which is the cloud software archive.
Those standards allow the delivery of software between the vendors and the operators that wish to deploy them. This is opening door to a lot of multi-vendor continuum delivery because of course each vendor will have its N CTD pipeline and each vendor will be delivering software to us on a much higher cadence they've seen before. Also there was a new shift with this delivery to move from project to products. And for the first time, and specifically in my squad, we started to build out a product and I think it was a real eye opener for the operations team to understand that the stuff we were building, we would also be supporting let's just wind back a bit. What is a packet core when you use your mobile phone over the wireless network, or should I say the radio access network or your data and all the control is done by what we call the core network within that core network, there's something called the packet core. That's basically set up to control access to the internet, how much data flow and provide quality of service. The packet core basically is your pipe to the internet.
Move back to the XY story. FC standards were pretty much ratified. It was actually identified very early on that C I C D would exist. And you can see here by the diagram, that's, um, a part of the XY standard there that actually you will have multiple providers. Each provider will have their own development and test execution cycles. The idea here is, is that C I C D and DevOps principles are kind of built into the industry. There is the identification of a validation and operator stage actually for us side and for the sort of real world experience, the validation operator, kind of the same thing. We always test the software before it goes into production. The trick here of course is we need to make that fully automatic.
So the question is how can we cope with multiple vendors coming through? How can we cope with multiple vendors delivering that software? We knew we had to build a high level framework to help that delivery together with the squad. And after all our understandings and inspirations and readings, we slightly started to piece together what our pipeline will look like. This would be our continual delivery pipeline. This would be the, the high level north star to which we wanted to deploy. You can see here, the various phases we wish to break down ingestion, scanning validation, an independent testing phase, followed by a promotion into production. Then with the deployment and a monitoring classic phases of continual delivery subtly within the validation, we have our own vendors supplying our own software pipeline tooling as well. Big vendors are coming along with their own C I C D type tool sets because they're expecting to deliver an off the shelf capability.
Their capability allows automatic deployment of their software regardless of environment. So our level zero pipeline needed to interface with the vendor pipelines that they were delivering. Once the vendor have delivered their software, we needed then to do our own testing and validation classic phases, which could be run parallel, such as security, regression, integration, uh, load testing. And end-to-end testing will all take place. A lot of the tooling now in our test beds and labs are fully automated and we are expediating, um, the creation of new tools to help support this activity. One of the biggest things about the pipeline as you know, anyone that's worked in continued delivery is actually the shifting of left of security and controls because under normal delivery circumstances and the old way of working security was always left to the last once the system was stood up and built and ratified as working security were then rolled out to check that the system was actually secure. This means that a number of things could have gone wrong. This means that the project could be late and that actually security measures or, or issues or defects that were found may have to be waved to keep the project on schedule by shifting security left, uh, we've been able to actually mitigate bad software or bad defects from even entering our network at all.
Moreover, what we developed here was a single framework allowing for delivery management and the visibility of all the software coming into our network, but most critically, and a lot of people missed this, especially vendors was actually generating a rapid feedback mechanism. We need to get back to our suppliers, the fact that there's a problem so they can react. They can turn around the software, fixes the fastest possible. They can turn around the security fixes quicker. So how did our delivery pipeline involve chapter six, lean for us? This was in aha moment. We'd figured out via various proof of value experiments within our own sample, that there are certain mechanisms that would help us specifically the scanning and the security side of things. We would always brainstorm every week about different possibilities and how we could set things up. But the penny finally dropped for us when we realized there needs to be an MVP, what can we deliver to the business that would add value right now?
We decided to make that a very simple scanning and deployment, uh, sorry. A scanning and delivery mechanism software has to come into the company. Everything was set up to do the scanning and to try the supports and get the acquired sign off to allow that ingested software to be used internal. So that's what we made our MVP. We invited our vendor along to help build us the capability. They took part in our squad as a squad member. And that was invaluable because they saw day to day our, our troubles. They were able to guide us as theorists and also input to the development part as well. We used the show Intel sessions of the scrum ceremonies to show our stakeholders where we were going to start with. It came very obvious that lots of people were very interested, but as the sprints moved on, the audience started to shrink.
The show entails, become more of a kind of look at what we've done to ourselves. So it took a while for the share the stakeholders, really to really engage and remind them, you know, remind them that they have to be there to see what we're doing. Our first MVP demo was absolutely awesome. We were able to invite a member of the operational directorship or leadership team to come and operate a software delivery pipeline themselves with minimal information. They were able to kick off the pipeline, ingest the software, produce the scans, produce the reporting and allow that software to be promoted into a lab area. Ready for testing. Another aha moment for us was to put our own pipeline into continuous execution. Not quite continued delivery, continued delivery would mean we were running it quite frequently, but continuous execution and there's Astros there. We just started off lightly. We started to run our pipeline once a day overnight. We found out that continuous execution gave us visibility. The next time the pipeline broke, not the next time we needed to use it. This is a key differentiation.
This is where GitLab comes in again. And as they've been kind enough to invite me to speak on their behalf, a little section about how they enabled my team to develop. So I had the GitLab environment, which we were able to stand up quite quickly using Docker container. We were able to set up a rapid development environment. We were able to link, make sure our code was kind of secure. Do unit testing and build and push our actual deployment stuff into our artifact. Repositories the GitLab dev DevOps platform really sped us up. These of use of Goey and the rest of it allowed us to coordinate the team and set up people into our sound environment, very rapidly using the rapid CI development cycle. We could quickly get feedback to the developer if there are any problems, if the code wasn't satisfactory, if it didn't pass the test, if they'd left the password in the, in the file somewhere, this allowed for quick collaboration, immediate deployment as the build artifact were instantly available, should I say successful build artifact were instantly available to our own pipe, our own delivery pipeline.
We also used the output of the builds, uh, for code health status. We knew if the code was clean, if the code conformed, and if the code was secure contribution statistics from the GitLab engine allowed us to analyze squad health as well. We don't believe in marking squad velocity via lines of code. No, this is more just to analyze how the squad was collaborating. Are certain contributors doing more than others? Does someone need some help? Have there been specifically specific sprints or parts of the problems or we've been solving that generate lots more code? It was a fascinating insight.
Moving forward. We can look at the outcomes, all this work and all this effort. A year later, when we started up our squad, we were told that it could be to take around two years for transformation. It could take two years for our way of working to really be in and for the management to understand and see what we were capable of because on the traditional waterfall path, the Target's already been set. Whereas with agile, we were finding our way that was very difficult for management to stomach. There was quite a bit of resistance or misunderstanding of what it is we were doing. No idea what it is we were achieving. We knew what our sort of goals were. We knew we had to build our framework and we knew roughly how we wanted to do it. So we were drawing the rough outline of what it was we were gonna deliver. It's just without those concrete deadlines and maybe pause that the management were a bit skeptical about what it was we were gonna achieve. In one comment, we were very showy in telling how different that is a year later.
Now we've worked as a team. We've managed to deliver a new security framework. The business is really starting to open up to not only are we passing our container solutions via, but we're also starting to pass quite a lot of our virtualization. Um, VNF solutions via we're able to deliver SPO level visibility that enables rapid tracking of problems and faults and security holes. We've been able to build out. And we are building out rapid dashboards. This information is readily available to software passes through our pipeline. We are able to ingest it, pull out metadata, pull out the re uh, release notes, pull out the testing information. We're able to scan for changes. We can throw up anything that's not looking right and immediately find it
Another outcome, which has been inherent to the adoption of DevOps practices via the use of infrastructure as code and continuing environment. Health checks means we've been able to improve stability of environments no longer were people just going in and changing anything. Willy nearly even though the control mechanisms were in place in our test environments, we still have vendor engineers coming in, making subtle changes. We still had, you know, J F D I situations happening. They're all hallmark situations of an old, you know, an old way of thinking. Introducing infrastructure as code and version controlled configuration meant that we knew exactly what should be deployed. And if anyone had tinkered with that, it was clear what was changed. Deterministic environments are key for stability and observation. The continuous health checks gave us the confidence that the environment was safe to move forward with critical testing, just to summarize some of the aha moments, the best I can, as you can imagine, it's been an interesting year and a half of experience.
I can tell you now, one thing that opened my eyes with the dome, where Fin's pointed out quite validly, it's not always about the dome. There's tons more metrics that support the door metrics, but those door metrics are actually a north star. If you can look towards delivering them or at least make a start, you're heading in the right direction, quite simply put the facts, don't lie. Every effort's been made by Dora to analyze the state of DevOps and how it actually affects companies and how it does impact high performance companies. I also like to call out this, this summit, the DevOps enterprise summit is one of the best summits to be inspired by lots of fantastic speakers, lots of fantastic people, and also authors and material coming out.
And when we learned about agile and me, we talk about small batch cross-functional teams, Conway law. It's all true. You might have heard of these things. You might have discussed them, but I can tell from experience now, looking at these ways of implementing and looking at these ways of functioning, it really expediate the outcome. I could talk another hour about Conway's law, but for sure, the way we structured our squads and the way we structured our architecture had massive impact to how we delivered another simple one here and a little acknowledgement to a work in progress. Less is more be aware of the cognitive load on the teams. You've got a squad full of smart people. They think they can boil the ocean. The moment we started restricting our sprints to one or two items, the more we got done and the more broader we were able to look at the solution. And lastly, I think it's true exactly as the Phoenix projects and the unicorn project state joy can be brought back into software delivery and development. So thank you for listening to me. I hope you've enjoyed the talk. Feel free to drop by the bar. We should be there. And if not, maybe catch up for you on a community event. Thank you.