A Trail of Breadcrumbs: Virgin Atlantic Airways' Journey Towards Agility

This isn't a talk about...


-how we've got it nailed at Virgin Atlantic

-Puppet or Jenkins or Azure DevOps or Teams vs Slack or how we only use perfectly-chiseled T-shaped SREs or The Golden Framework etc. etc.

-the right way to "do it"


This is a talk about...


-what to do when you've been told to fix the un-fixable

-what to do when you don't know where to start

-what to do when what you're doing doesn't seem to be working

MM

Michelle Moss

Technical Release and Delivery Manager, Virgin Atlantic Airways

NW

Nic Whittaker

Head of Platform Engineering & DevOps, Virgin Atlantic Airways

Transcript

00:00:07

Welcome to a trail of breadcrumbs, Virgin Atlantic airways journey towards agility. My name is Nick Whittaker. Hello, gorgeous.

00:00:25

So who are we? Well matter to us. If you're flying on a budget or an upper-class, we will make you feel the most loved we'll do with Virgin flair, whoever you are and wherever you're going. Um, so we fly passenger planes and cargo planes to a little over 30 destinations, and we also do package holidays. Um, prior to the COVID situation, um, we were reporting a revenue of just shy of 3 billion, which is, uh, an improvement on previous years, um, uh, directionally, um, and, uh, uh, year on year increase in passenger numbers, uh, post COVID. Well, you'll have seen the news. We're not flying passenger planes at the moment. Although the good news is we're planning to restart next month to a small number of destinations. Um, however, on the upside, this has been our busiest cargo period on record Michelle.

00:01:17

So we have around now 9,000 employees that this is now being reduced by third, with around 90% of us furloughed and within technology, there were 220 with a heavy supply chain. And this is now being reduced to 165. Our structure within technology is we have our CIO who runs two, has a VP of transformation. Nick reports into her. And then along with testing dev and dev ops, I report into Nick, Nick,

00:01:54

Sorry, what's this talk about? Well, I've been to many of these events. I've sat where you're sat, not at home, obviously in order to rehab, I've seen companies like added ass Barclays Jaguar on the stage explaining how dev ops they were, how they had these cool pipelines that were deploying billions of times a second. It's production generating huge value for customers and internal value for the organization. And I'd sit there where you are. And I think, well, that's all very well and good, but how do I get from where I am to where you are? Um, I've got no funding. I've got no buy-in um, and it sounds a bit like what you've been challenged with is a standard three where a three tier web stack I've actually got an eight year old episodic flat file mainframe at the backend. Um, I've got Oracle servers, I've got BizTalk, I've got MPQ.

00:02:45

Um, I've got, I've got AWS. I've got no idea where to start. So this is our attempt, um, to, to describe how you can navigate that journey. Um, what it isn't is to talk about how we've got it nailed at Virgin Atlantic. We haven't run the journey, but there's a lot of work to do. Um, it's absolutely not, uh, talk about what, uh, tools or frameworks you can buy that will make you agile or make you DevOps. Um, and it's absolutely not talking about the right way to do things, but hopefully we'll be able to impart some lessons as to how we did it. Michelle.

00:03:22

It is a talk, however, about what to do when you're asked to fix something that is fundamentally broken and you've got no idea where to start from and what you are doing, doesn't seem to be working and given current circumstances, we are also going to touch on what happens when big things go wrong, Nick.

00:03:45

So how did the journey start? Well, it started with a reorg and my boss at the time saying to me, Nick, we need some devils. And he gave me this pithy job description, lots of work words. Um, after I'd assimilated it, I turned around to my boss and I said, all right, how are you expecting me to do this? Um, his response was pretty much, uh, figure it out. So once I'd calm down, we asked the question of the first breadcrumb, why is it? We were really having the conversation or what is it that my boss was really getting at Michelle,

00:04:25

The problems we were having where that's Virgin Atlantic was heavily siloed. We ran within divisions. We had technology within commercial. We had a digital team with their own tech people and operations with their own technology being delivered. It was leading to lots of conversations that sounded like we want a website. Didn't, you know, we've got one already, Nick.

00:04:50

Uh, we had lots of technology. We had all the technology, as I said, we are the back end. We've got big 80 year old TPF mainframes. Um, we had a huge rate, a range of middleware. Um, we had some nice in cloud technology. We had a number of websites, um, and we, we had, um, you know, uh, very complex technical, uh, architecture, Michelle,

00:05:14

Um, whilst you wouldn't expect Virgin planes to have an EMS, We did have different projects in different silos. Delivering over are overlapping things, tangentially. We own all of the CRMs and we use 10% of each of them, Nick

00:05:36

Business, often didn't trust technology. So golf with a credit card and delivery solutions. First thing we would hear would be a call to the help. They're saying quick, our mission critical Smartsheet application at the airport has gone down. Can you fix it to which we'd reply? Uh, we'd like to use Smartsheet, uh, w w we're licensed for power BI. What do you mean? We can't land planes, Michelle,

00:05:58

To make it worse. Leadership would disappear off site once a year where the bag of bones and runes and after their midst go dancing would come back with a pronouncement of what we'd be delivering this year to make it better. Do you think we ever over-delivered an unspent on all of this stuff, Nick,

00:06:21

Finally, I had a growing awareness that it might be a better way to do this stuff,

00:06:27

Michelle.

00:06:29

So our second breadcrumb, what did we need to do? Are our silos wanted more from us than we could deliver. We had people using the word agile a lot. We were getting people external training, receiving their scrum certificates and coming back to run waterfall projects. We were spending money on large projects, which were running for ages and delivering something. We didn't have a day, one starter pack, but we did have a nice textbook pictures about our process. We also had a lot of high cost contractors within for, in their heads, but nothing written down, uh, or reusable, like

00:07:12

We didn't have a single source of governance truth. Um, what we did have was a huge landscape of suppliers, um, creating, testing our code and supporting it in short, it felt that we were reinventing the wheel for any new project and a little of it was under central control. So what might fix the problem? Well, people were banding around the word DevOps, give us some dev ops in technology and that will sort it out, um, that will make us more agile. Uh, we had absolutely no understanding in the organization of hidden work or the cost of handing off between complex processes. People were using the word speed I wanted faster, but did that mean they wanted the same thing for the same money only sooner? Or did it mean they want to just see some incremental value, uh, arriving? It was becoming apparent that DevOps wasn't just about a conveyor belt, a technical conveyor belt. Um, but with something to do with the kind of collaborative behaviors that we were not seeing at this point across the organization,

00:08:17

Uh,

00:08:18

Meanwhile people were asking the questions, but so what, why did they want this stuff faster with more agility? So the third breadcrumb, what did we have to work with? Well,

00:08:31

We had some great partners, um, air France, KLM, uh, Delta airlines, um, were both on their own journey. Um, air France, KLM, for example, um, we're into venue way of working, um, and they were rolling out safe. So we had some conversations there. Our partners at Delta had just started their own agile dojo. Um, and they were mandating that any projects that delivered into their new cloud stack had to go through training through dojo. So it was the lessons we could Crip there. Meanwhile, I was having conversations with external organizations, um, for example, uh, Tom Clark friend of this conference taught me everything I needed to know about culture and people. Um,

00:09:12

Meanwhile, we had a lot of partners, um, in terms of suppliers, but were any of them willing to step up and be delivery partners as it happened at this point in time, one of our suppliers was undergoing a con contract renegotiation that gave us the opportunity for a bit of quid pro quo. Could we give them something in return for an alignment on agility and some free dev ops resources turned out? We could. Um, we also looked at moving, some of those project costs around. Um, and also we focused on that training budget rather than sending people out Virgin Atlantic. We've got classrooms where we train crew. So why didn't we use that environment to help align people on, uh, agile best practices? Um, the fourth breadcrumb, um, we just had to start to remind you have no money sketchy mandate and no top down buy-in, but start, we did

00:10:07

Turned out. We had one project that actually could lay claim to looking a bit agile. Um, it was, uh, called, uh, crew apps. It was delivering tablets to our flight crew, uh, had to iterate a lot. The crew were exact exhibiting, really mature agile behaviors. However, they were being funded as a project, a value that they would soon run out of runway when that project funded stopped. And they had to hand over this living set of products into a price, into a legacy operational support mode. Meanwhile, we had a couple of other projects that were starting up. We had a data insights project. We had an external API project. We had that holy grail of projects, the single customer view, all of these things required, data governance, API governance, technical governance, they're all deploying to cloud and they're all going to iterate. So we decided to get our arms around those initiatives. Meanwhile, we brought some external agencies in to give us a benchmark of our own maturity. We also ran over a dozen workshops across the different silos in the organization to figure out what we did well and what we didn't do well. Um, and we also began doing value stream mapping across some of those processes to begin to understand the bottlenecks and the hidden work and all that kind of stuff.

00:11:26

Michelle.

00:11:28

So fifth breadcrumb we're taking off, but how are we going to generate thrust? So we were looking at partners and they were using assessments and certification within their teams. And this had led to increased autonomy to improve their team's capability. And we wanted some of that too. We then develop this for, to give our teams ownership if they improve capability and to then increase that autonomy. Part of the output here was looking at ecosystems and the ops processes into them where we're building CIC D that then needed several days of change management process to reach prod. We were then tackling these bottlenecks teams, which are generating their own values, which over time then converged across the teams. This is about people first tool, second, getting ourselves closer to a high trust, low blame culture.

00:12:29

So we were now fostering engagement, bring in our naysayers. We set out to do a series of educational. One-on-ones getting a sense of fun into the work and bring in several areas and partners. This resulted in a raft of resources that we could then draw on. As we built up the teams, we had been flagged by our partners. That's curious, security was a key area to have onboard early in doing this. They supported champions in our teams and ended up being our as partners from ops. Um, this then started encouraging other communities to start to fall Nick.

00:13:12

So the sixth breadcrumb, uh, this point in any good narrative, the middle act, you begin to hit the jeopardy in our case, the problems, what were those problems? Well to a mania, they started mounting up.

00:13:26

Yeah,

00:13:30

We had sponsors, um, who decided that it was taking too long to get the DevOpsy quick delivery that they wanted and it was costing them money. Now at this point, um, we were beginning to generate hard data from, um, our capability assessments. Um, and so we were able to use that, to demonstrate that the focus on our teams was the right focus and that actually it was yielding a sprint upon sprint improvement in things like quality, um, and addressing technical debt, um, and improving turnaround times, um, and, uh, and realizing, um, some kind of value from the stuff that has been delivered. And we were able to bring those people back on board. Um, we're also trying to have conversations with legacy areas in ops, like infrastructure who would say, yeah, we'd love to help. It's a great idea, but look at our work load. You're going to have to give us more people just to get through this work, like before we can even learn to be agile. Um, There's always hidden costs. What do you mean? We need more MSDN licenses. Um, every time we spin up a new agile team, it would have to go into a new room. On the other side of the building, we were just building even more silos.

00:14:39

Uh, it was taking weeks just to onboard staff just to get them a land yard into the building. Um, meanwhile, um, those first resources who had brought in from our partners or poached by external consultancies selling DevOps. So the one thing that we were learning at this point in time was the toughest lesson really around resilient Michelle,

00:15:07

But we've made real progress. When we stopped to look back over our shoulders, we saw that we'd moved much further than we had thought. We're working with team daily, Tim Crawley, and create our offshore capability. And I had this explicitly as part of my job description. And we began seeing our offshore teams setting up their own cam Kanban's and addressing the bottlenecks. We were working on core platform teams to help delivery teams be more autonomous. We're creating our dojo to optimize, collaborate, and create the best outcomes for those teams. We're moving from JIRA and confluence, QC arm, and bringing them and all of our new projects into Azure DevOps. So we were starting to increase our process and testing automation as part of this and with the capability being owned by the teams, um, champions from teams began meeting as communities, especially our product owners, deciding how they were going to generate the value that they wanted. Nick.

00:16:11

So why are we bothering with this again? Well, um, we now have a groundswell of people who, who were getting it. I may work collectively be an in to understand the need to move away from, um, siloed projects to products on platforms built and run by core capability teams. And we were collectively beginning to understand the need to think about this idea of value streams and outcome based thinking,

00:16:37

Which takes us to the seventh breadcrumb, a change at the C-suite I've been at Virgin Atlantic for around six years. I started in 2014. At that point in time, we had a new, a new CFO. I had changes in the tech leadership, 2015, brought a new interim CIO and an accompanying reorg in technology, 2016, yet another CIO and another reorg in technology 2017. I need you CCO and your CFO and a VP for business systems, VP for ops and another reorg in 2018. Another new VP in technology and 2019, we've got a new CEO, a new suite, including any CIO. I didn't EVP for transformation. Um, and guess what came with that? Another reorg,

00:17:24

However, the new CIO CEO brought with him a new C CIO in it. And, um, this individual had an agile background and also crucially a growth mindset. Um, he and his team understood the need to view all change in a common way from a single place to reduce the technique technical landscape, to be into target core capability and remove what didn't actively support the vision and to increase the, uh, responsibility and repeatability and quality of output from teams, which is really a lot of the work that we'd been doing up until that point in time. Michelle.

00:18:03

So the CEO now had a very clear why to be the most trot loved to travel company and then technology. We had the same vision to be the most loved technology against which our work was now aligned. So we then bought this into our teams, weathering the things that were outside of this vision. We started delivering the right things to bring the value early. This was supported by our amazing delivery partners to help us achieve this vision, Nick.

00:18:38

So we now I'd always honest from the top down. Um, we'd learn how to do repeatability at a basic level. Um, we knew we wanted to capabilities and we knew we needed to reduce divergence and waste. Um, we knew we needed a single routine for all demand. Um, and we were beginning to talk about value streams and portfolios. In short, we were approaching a critical mindset change at organization level, um, which takes us to the eighth breadcrumb, What you can do when you've got that top down. Buy-in Well, now we were approaching a big cultural shift at enterprise level. We set about changing the way that we thought about change. We established a bunch of core value streams, and each of those was owned by C-suite executive sponsor, Everything that was going into those value streams, what was being proposed for absorption into those streams was being scored against the Y using our corporate measures. Um, we insisted on visibility and discussion between those executives across those streams, talking about dependencies and risk and the cost of doing it and the risk of not doing it. Um, we ensure that we had representation from everybody involved, um, who might be accountable in the delivery chain. So change, run, finance, procurement, HR, uh, these people were all co-opted.

00:20:06

Um, we synchronized everybody to a quarterly planning cycle. Um, the enterprise PMO was brought from it's a rich funeral home in finance, um, near to the change delivery and that allowed them to have good governance at studio level, um, and good reporting, um, across the whole piece. And that allowed us to get to a single source of the truth. Um, so what were some of the lessons we learned during this critical period?

00:20:36

Well,

00:20:38

We found that if we refer to the historical fact that we couldn't let off demand versus budget versus resource versus outcome, um, that gave us the impetus to move to a new model. We needed to show that there was a model that was externally relied upon something that had been proven. Um, okay. So we start over safe. Uh, we used baseline scoring against our well-established corporate measure measures also factory in risk, not to do so ROI and all those kinds of things. Um, so when I have to deal with conflict in terms of prioritization, we mandated that anything coming in, any investment proposal had to use that single funnel. Um, we also had folk all across the data and we dashboarded everything and made it visible all the way up to the CEO on a minute by minute basis using our common, um, forward looking strategic tools. Uh, and in our case we're a Microsoft shop and that's what we were focused on. Um, and during the events accompanying this, we filmed them all doing it. Uh, we put the leadership team in the middle of the room to resolve conflict in front of the entire audience.

00:21:54

Repeat. So in summary, we now had a new way of working. Um, we had a single plan for all the demand that was going into the organization, broken down into value streams that had visibility and dependency at the core of their thinking that could be broken down into programs and its delivery teams who were already thinking of things in terms of iterations, uh, Kanban, scrum boards, and so on. Um, that's worth noting that at this point in time, the teams at whispered up with agility and DevOps, we're probably only 20% of what was being delivered. Everything else was still waterfall. However, even those waterfall teams, we're still leveraging the best practice things like test automation, um, things like, um, uh, uh, stating out there value, um, and reporting those metrics back up on a bi-weekly basis into the dashboard. So we were beginning to see progress. Did it work well, see for yourself,

00:22:58

Let's start this new decade. We're having to think differently about how we prioritize and change. And what we're seeing is organizations having much more interlinking capability in what they're doing in their priorities and their strategy. So today the pie is about for the first time ever. It's a first for us, everyone in this organization, sharing their assumptions and their planning assumptions to one outcome around velocity. One of the things I really believe in is a growth mindset. So should I talk about growth all the time and we will be profitable this year, but to, for us to remain profitable and to continue growing our organization, we have to have this growth mindset to investment, to planning and to execution. And this is what the PR is really about. It's about having a growth mindset to adapt, to adopt. And so it's all new ways of working.

00:23:50

The first purpose is really to decide what's important for the organization. What will deliver us the most value, the most benefits, the most risk avoidance. But the second purpose is also to get to a particular number. We have a cost envelope that we have to hit, which is 35 million of CapEx this year, which has been decided it's not going to go creation. And so we need to make sure whatever we think is important fits within that 35 million

00:24:10

I'm most concerned that will come out at the end of the day. And the number won't be 51 million. It's going to go up again. Everybody started adding things back in. We went from 35 to 40 to 48 to 51 overnight, which surprised me this morning. The thing that makes me most excited is that we do bring it back down and people start to be the real leaders of the organization. If we get this plan right. And we deliver the things that deliver the real value, we all deliver our velocity plan. So we all win.

00:24:40

It's intense and it's noisy and it's quite hard. So yesterday obviously was the first day. So none of us really knew what to expect. So it's quite hard to work out exactly. But today I would say we're a lot more, I think we're getting there.

00:24:50

We are a day and a bit in, within three hours in it yesterday afternoon, we close the gap that it takes most organizations at least a year to 18 months to achieve. So from my perspective, how's it going? It's going for a nominally. Well,

00:25:04

You've got a whole sort of cross section of the company to come in and really understand what their projects are, what they mean, the value of the projects and the cost of those projects and how we prioritize those projects. We've got a huge amount of demand and a small pot of money. People, the business partners here, you've got the SMEs here. You've got the budget owners here and it's making some really challenging, some really interesting conversations.

00:25:26

We have a lot of finance representation. This room is just fantastic given our team actually because of the opportunity to showcase their skills and how they challenge, influence and make sound financial decisions. On behalf of our business,

00:25:38

There's been a lot of collaborations, have conversations and really understand what we're trying to do across the whole business, which I've never seen before. So it's really good to understand audit dependencies on each other and it purchase that maybe we didn't know about how they might integrate of our projects. There's lots of exciting stuff to come down the line. The difficulty really now is it's because there's so much great, exciting stuff is really trying to prioritize what we need to work through as to what's really going to deliver the most value this year. And we've been here where we can,

00:26:06

We have a very ambitious company. There's always more that we want to do, you know, or is it bigger than all? Barely I think a lot of the time, but I think the most valuable thing that's come out of this is the conversations and the quality of the conversations I've had in the room.

00:26:17

Well, I've learned, don't underestimate the culture of Virgin Atlantic. There are very few companies in the world that can go from an old way of prioritization and enterprise. So the most modern ways of thinking in agility, we can do it and not only have we done it, we've done it brilliantly and come out of it as still as friends.

00:26:41

So I think at this point, it's worth noting that you heard some numbers in that video, those numbers were all pre COVID. The world has now changed. Those numbers. Don't apply. Nonetheless, the outcomes do apply. Um, and we're going to talk about those now. So we went into that event with, uh, if you like it, an MVP, Number one, we needed everybody in the organization to understand that this new methodology was how are we going to do it from today onwards when we need everybody to understand their part in that chain, We had a challenge. The demand at that point in time on the, on the first minute of the first morning was over 51 million and rising. We had to bring it into a 35 million envelope and we had to do that within a headcount constraint. And finally, we needed to get commitment that we would do this together today and going forward, uh, on the, on the premise that the silo discreet way of working previously had not worked. How did we get on Everyone understood their roles you saw in the video, every got on like a house on fire. Yes, it was difficult, but we got through it.

00:27:59

We hit the numbers. We went from 51 upwards to nearly 60. Um, we pressured everybody. Um, we held everybody to account. We make the leadership speak in the center of the room. And by the end of the event, we came in at 35 million and on head counts And we got my commitment and there was nothing stronger than one of the C-suite. You came up to me and said, Nick, I never thought it would work. You changed my mind. So that was a win Michelle.

00:28:37

So out of breadcrumbs, just when you think things are going well, you may have noticed that lots of things have gone sideways sideways recently. Um, we've been hit with many challenges all at once. You may have seen in the press that globally passengers have stopped flying cash preservation has become our number one focus. Um, we're also managing rescuing customers, refunds, postponing bookings, and starting up cargo, only flights to ship PPA for the NHS. Um, huge impact was felt by all of our people, but where the culture that has really shown through Nick

00:29:23

There is an upside. Um, we have had to learn rapid, uh, adapt.

00:29:32

Um, we've had to move legacy teams away from, um, stating things in terms of complexity, towards a view of iterating on the complex. A good of this is, um, the infrastructure team, um, that I, I alluded to earlier. Um, they had a two year plan to roll out Microsoft teams across our, our organization. It was very stepwise. Um, and, uh, you know, uh, there was a lot of change that had to be managed during that period. However, we were suddenly in a, in a period where we were furloughing huge amounts of staff and those staff who were, uh, who were remaining at work, we're going to have work remote like that to your teams. Rollout happened in two weeks from a legacy thinking team. Who've never changed their mindset. Um, we've also had, uh, our teams having to disperse and work off shore, uh, really learn how to coordinate, um, and, and work highly collaboratively, Michelle.

00:30:35

Um, and there is stuff that we can do our skeleton team focusing on maintenance and governance that we often neglected and we're giving our standardized tools up and running, running for when we do restart our passenger flying at the end of the summer, Nick.

00:30:52

So there are still obstacles ahead. Um, I'd probably help, um, needed, um, staff reductions, um, and, um, the reduction, the huge reduction, apple folio, um, have obviously I mentioned, we're starting flying again in the next few weeks, buy tickets and holidays please. Uh, more seriously. Um, we are focusing now on those more legacy, uh, teams, infrastructure teams, and so on, uh, beginning to help them in the way that we've helped have change delivery teams. Um, I think the next big step for us, um, and the area that we need support and help, um, is around revising and improving that first pass that we took at valley streams. Um, it was a good start. It was, uh, it was a mechanism to allow us to get the C-suite to work together. Um, however, there not yet true value streams, a number of those value streams are measuring and reporting on the same, uh, same values. Um, so that's going to be a work in progress. Um, and I haven't dropped my mouse. Let me just pick it up. Um, I'll just leave you with the thought that during this challenge, um, we still have to constantly adapt and revise and change our way of thinking. And these are all, um, lessons skills, values that we as an organization

00:32:10

Over the life of this journey. Thank you very much.

00:32:14

Thank you.