Amsterdam 2023

Disney Global SRE – Creating Digital Magic

Let us tell you a story about a century old organization that has scaled its SRE practice to ignite digital magic across the globe. This team of SRE Jedi Knights are on a mission to foster curiosity, communities of practice and technology awesomeness while venturing where no SRE has gone before. In this talk, we will deliver epic stories of successes, setbacks and failures while pushing large scale platforms to their limit and delivering the best in-seat, digital experiences, products and content to our guests and subscribers across the globe.

JC

Jason Cox

Director, Global SRE, The Walt Disney Company

Chapters

Full transcript

The complete talk, organized by section.

Host Intro (Gene Kim)

[00:00:13.060] All right. I am so excited that our first speaker this morning is Jason Cox, Director of Platforms and SRE at The Walt Disney Company.

[00:00:21.460] I met Jason over a decade ago, and I remember so vividly when I finally got a chance to follow him around for an entire day. It was 2014. We were at the Starbucks on the Disney Glendale campus, and he was walking me through his calendar, his diary, his agenda, describing what his day was going to be like.

[00:00:37.660] As he was doing this, a woman across the room saw him, ran over to him, and said, thank you, thank you, thank you, Jason, so much for saving our team two weeks ago. He later told me that she was a CTO of one of their business units.

[00:00:53.160] That day I got to see so many amazing things, but one of the most remarkable things was that I got to see that little scene played out again and again. In fact, I think I got to see him thanked more times in that one day than most ops practitioners get thanked during their whole careers.

[00:01:08.990] Over the years, Jason's presentations have inspired so many ops leaders to rethink how they are organized, including Fernando Cornago, now VP of Digital at Adidas, who will be talking next. Jason spoke at the very first DevOps Enterprise Summit in 2014. It was memorable for so many reasons. One was that he played previews of the then-upcoming Star Wars movies, and another was that his presentation was not allowed to be recorded or live streamed for enterprise reasons, which I am sure you are familiar with.

[00:01:37.580] I am still delighted that, thanks to Jason, Disney now allows their world-class technologists to share their stories, reinforcing the amazing brand of engineering excellence that enables the magic that Disney is so famous for. Here to share where Jason's journey has taken him is Jason.

Jason Cox

[00:02:03.230] Good to see you all. It is good to be back in Europe in person. How wonderful it is to be here.

[00:02:11.740] One of the things I love about working for Disney is all of you. Wherever I go, as soon as somebody finds out I work for Disney, they smile. This happens over and over again. They begin to tell me about their experience with Disney, our content, the thing that inspired the home, the opportunity they had to take their family, their friends, their loved ones to go see Disney World, Disneyland Paris, wherever. I love that.

[00:02:52.540] I love to be part of a company that is all about them. Our mission is to inspire, to educate, to entertain the world, our human family, through great storytelling. In fact, our founder Walt Disney, that was his dream. He dreamed of that. He wanted to create delight, create happiness across the globe, and I get to see that in the faces of all of you many times.

[00:03:23.430] We have just reached our hundredth year, and we are celebrating a hundred years of doing just that. We put together a video that talks about that. Watch this.

Disney 100 Video

[00:03:38.480] Open your eyes.

[00:03:43.740] Think of the one place you have always wanted to see.

[00:03:51.320] This is where magic comes from.

[00:03:56.140] Dream more than once. Please, please, please. It is sure to come true.

[00:04:03.220] Everything you see exists together in the greatest circle of life.

[00:04:13.360] It is true, all of it.

[00:04:17.470] In every job that must be done, there is an element of fun.

[00:04:25.760] Life is full of possibilities. What do you want to be remembered for?

[00:04:33.870] It is in our hands, and we have got to do something with it.

[00:04:40.460] I just want to say a word to all the workers, and all of you, and the young at heart. You will help make this dream come true.

Jason Cox

[00:05:07.410] Thank you. What do you think? I have to admit, I do get emotional sometimes watching that. This is what we do as a company. I want to talk to you about how my team, our SRE team, is helping deliver that magic to help those dreams in fact come true.

[00:05:30.860] I am going to talk to you about three different lessons that I have learned as part of that journey at Disney.

[00:05:41.670] First of all, where do I sit? This is the weird org chart you could start every meeting with. It gives you an idea of what we support across Disney. We have that amazing storytelling powerhouse of the company. On the left are Disney Entertainment teams that are really crafting and creating that content. We have ESPN, our sports division, delivering those experiences and that content to our viewers. When we turn all that storytelling into concrete places you can go, destinations you can see, they are on the right through our parks, experiences, and products.

[00:06:22.760] Here is my group, this big green bar at the bottom. We are corporate. We are here to support all these different amazing brands, deliver that magic to our human family across the globe.

[00:06:38.990] There are problems with centralized services like ours. How many of you are part of a shared service team in your company? Okay, I do have family here. How many of you use shared services in your organization? How is that going for you? You have suggestions?

[00:07:18.120] Here is some of the problem. A lot of times shared services show up like this: we are here to help. You have your Star Destroyer, and they are here to help. Unfortunately, it is more like this from the businesses: help you do not. Of course, I like what Grogu says. The problem is, I do not understand it. It does not matter. He is so darn cute. But it is the same thing: it is not helping.

[00:07:55.910] What does work? What model would work for us? For the last several years, decades in fact for me, I was looking for that model and began to evolve my team into providing shared services.

[00:08:14.180] How did we do it? What was the model that began to work for us? It looked a little like this. We wanted to begin to embed our engineering teams, our shared service, our SRE teams, into all the different areas across the company. Embedded into the product teams, the business teams, part of those engineers that are helping shape, create, and deliver those products and experiences across the company.

[00:08:43.910] I know what you are thinking. This looks a little bit like a Death Star. That was not on purpose. I gave part of this presentation to Lucasfilm, and they told me the same thing. But that is not the only thing they said. They also said, Jason, we want you to know something: you are not like other shared services. Your team actually understands our business.

[00:09:22.580] I began to hear that over and over again as I talked to the different technology leaders at Disney. They said, your team is different. They would say things like, your team is valued. You do a great job for us. Thank you. We could not do it without you. Thanks, Jason. Your team is fantastic. Your team, Jason, is unbelievable.

[00:09:54.430] As a shared service, that should not be true. We should not be unbelievable. We should be doing things they actually need and that matter. But they said, thanks for stepping in. We appreciate your innovation and creativity for all that you have done to contribute to our team. You always have such a great attitude. Your team is innovative. You get the job done. Thanks for being an awesome partner.

[00:10:23.320] That is one of the lessons we began to learn. The lesson I will convey to you is: at the end of the day, start with this. Listen. Listen to the business.

[00:10:37.020] I asked some of our CTOs why we are different and what they see about other shared services that is not working. One of them told me, Jason, it is real simple. Nobody else listens to us. Would you go tell them: do not come and tell; come and listen before they tell. Understand our business. Understand how we ship the magic. What are we all about? What is our business about? Come and listen to us. Know what our North Star is. What is our objective? Where are we going? What is our mission? Nobody asks us. Would you tell them to come find that out?

[00:11:31.770] That is what I kept hearing. Know what we are all about. Know our mission. And they said: you also know our team, Jason, because you are part of us. You are a partner. You are with us. You are embedded in our teams. Know our team, know our culture, know our ways of working, and you resonate. You begin to add to that, and you are part of that.

[00:11:55.970] This is what we began to find out as we listened: what were all these different teams looking for? It distills down into helping them create content, products, and experiences. Better: higher quality, elevating quality. Faster: not recklessly, but delivering value sooner. Because then we can experiment, try new things, learn, and pull that in. Safer: protect our company, protect our treasures, protect our guest data. Happier: we are supposed to be shipping happiness, the dream Walt had. But happy cast members and happy employees deliver happy products. Create an environment that elevates happiness.

[00:13:02.410] Lesson number one, if you are taking notes: listen. Know the business, know the mission, know the team.

[00:13:10.650] Number two on the list: have empathy. Understand the frame of reference of the business you are supporting. If you are shared service-y, understand their struggles, their pain points, their problems. Put yourself into their shoes. It will help transfer us from the Star Destroyer that comes and tells you what to do to being part of the family.

[00:13:54.690] It is not creating ticket windows. If you need help, submit a ticket. How many of you know you work with shared services where the first thing they do is redirect? Where is your ticket, please? I am sorry, please get back in line. That is not helpful. Instead, go to them.

[00:14:17.470] Proximity-powered empathy engineering. We talk about that internally. That is what it is all about. You get to know that frame of reference, that context, for you to be able to deliver value to them. You have got to know them. That is driving that proximity, that location, and going to them to see.

[00:14:37.540] How does that look for us? We go on location. We support where we are. We help build the new things. We are there in the trenches where they are. We understand the pain points, the challenges, the things we are trying to do to deliver for the business to be successful, to reach our guests, to create that happiness, delight, content, and experiences across the planet.

[00:14:59.260] So have empathy. That is the lesson. Shared mission, shared struggles, shared wins.

[00:15:09.940] Lesson number three: actually help. How many of you have one of those shared services that do not help? I love what Ahsoka says here: do something to help them. If you do, people will come to expect that you can help them, and they will look forward to seeing you again.

[00:15:44.110] Reverse that for just a second. If the business does not look forward to seeing you, in fact they try to avoid you, you may not be helping. So try to help. It is super important. Actually help.

[00:16:05.890] For us, how do we do that? Let us talk about some of the ways that we help. Better: their goal is to increase quality. As SRE, we help define service levels. What do they need for the business? What do those look like? It is about the experience staying around for our guests. MTTR is important. Transaction times can be important. Businesses have metrics like dollars per minute. Those are important metrics. We should have service levels that we help define with the business and measure to those. When they breach, we actually go in and help restore. We build architectures that help drive reliability and those experiences.

[00:16:53.990] One of the managers of an engineering team in my group told me this. In art school, a professor leading a pottery class told half the class that their goal for the semester was to build the best, highest quality piece of pottery they possibly could, just one. The other half of the class had a different job: deliver the most number of pots. At the end of the semester, they would be graded on the kilograms of pottery they were able to produce.

[00:18:01.150] At the end of the semester, a curious discovery occurred. Guess which group had the highest quality? It was not the group focused on quality. It was the group focused on quantity. The first group spent all their time theorizing what good looked like, polishing to perfect, and ultimately ended up with a metric ton of theory and a dead lump of clay. The second group got to work building, learning from their mistakes, improving and perfecting their processes, delivering more until they had a higher quality product.

[00:18:50.520] When you think about better, think about how we drive better. If you are an artist and you are drawing, keep drawing. If you are writing, keep writing. It applies to us today in the software space too: keep shipping, keep building. Your release frequency increases the value of the release of the software. As frequency goes up, the performance levels I was talking about also increase. We see this over and over again. Part of our goal is to help drive that, so we can increase release frequency to drive quality.

[00:19:43.550] The other part of going faster is helping remove friction, the things that inhibit being able to deliver that value: the red tape, bureaucracy. How do you cut through that? We spend time figuring out ways to do that, and we join the Rebel Alliance. We become, in a different term, chaotic good.

[00:20:16.340] To help deliver faster, safer: security by design, shift left, put automated governance in place. I hear there is a great book on this, by the way. If you do not have a copy, it is called Investments Unlimited. If you do not know anything about it, go talk to John Willis or Topo.

[00:20:36.220] Happier: we ship happiness. How about bringing happiness to those who are creating these faster release cycles, these developers who are building? We want to build a community where they connect together, and we do exactly that. Creating self-service, low-friction ways for them to deliver; handing back time to our engineers; handing back time and empowering our artists to create these new experiences, these new characters, these new worlds that delight our guests and ultimately amplify our creativity across the company.

[00:21:22.570] As an example, when we were working on the Millennium Falcon, we helped the Imagineers build CI/CD so it delivers all the way out to the edge. You could sit in the ride vehicle, see a problem, open up your laptop, check out the code, fix what you wanted to change right there in the ride vehicle, check it back in, and on the next cycle you would see your code in place.

[00:21:48.540] What does that do? It allows you to see problems, experiment, perfect, and drive quality. We have done that over and over again. We did that again with Web Slingers. We have done that with our Galactic Starcruiser. What we heard over and over again from our parks was that this was the smoothest, easiest launch of any attraction they had ever seen. Why? It was not that we were new technology. No. We allowed them to see problems and then address them in a faster way.

[00:22:23.370] So actually help. Build trust, build community, build magic together. There you go. There are your findings. I am about two minutes over. You can take a screenshot if you did not get it.

[00:22:35.010] Here is the help I am looking for. What am I missing? It is probably ChatGPT, but I do not know. Come and tell me about shared service models that I need to look at, or SRE models that would be good for us to consider.

[00:22:48.270] Creating magic: there are really no secrets here about our approach. We keep moving forward, opening up new doors, doing new things, because we are curious, and curiosity keeps leading us down new paths. With that, thank you all.