Transformed! A Musical DevOps Journey with Forrest Brazeal
Transformed! A Musical DevOps Journey with Forrest Brazeal
Forrest Brazeal
Developer Communities, Google
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
[00:00:13] All right. I know exactly the moment in 2021 when Richard Seroter, whose work I've admired for many years, sent me this tweet. It was a tweet about this amazing song that the brilliant and talented Forrest Brazeal had composed and performed.
[00:00:27] I was just leaving the parking lot of a grocery store with all three of my kids. I pulled over and played it for them, and they loved it just as much as I did.
[00:00:38] So obviously, I then tried to find everything that Forrest Brazeal had ever created, and in my conversations with him, I discovered that we had a bunch of things in common, including an obsession with everything related to Hamilton and Lin-Manuel Miranda.
[00:00:52] And I must confess, in my grandest and wildest dreams, if there ever were to be a Phoenix Project musical, I'm hoping it would be as amazing as what Forrest has created.
[00:01:04] The other thing I learned about Forrest is that musicals are just one sliver of the amazing things that Forrest has done. In particular, I continue to be blown away by the success stories from an initiative he created called the Cloud Resume Challenge.
[00:01:19] So I asked Forrest if he'd be willing to talk about his amazing work and share some of his other amazing creations with us, and I'm so delighted that he said yes. Here's Forrest.
Forrest Brazeal
[00:01:42] Good evening.
[00:01:46] 5:28 p.m. Do you know where your digital transformation is?
[00:01:51] We know where it started, don't we? It always starts the same way. It starts with high hopes. It starts with big dreams. It starts with a vision.
[00:02:09] [singing] Picture us, glorious, rising up at last to be reborn with cloud, big data, and AI. We'll bid our legacy ways goodbye. We're caterpillars now, but we're gonna fly. Yeah, we're gonna be transformed digitally.
[00:02:35] Never done it before, but how hard could it be? All these slide decks look so great. Set the deadline; won't be long to wait till we declare victory digitally. How hard could it be?
[00:03:01] Years pass, alas, fighting still the fires we fought before. Our POCs looked great, so why are we not seeing ROI? And all the time our spend is climbing high. Oh, will we ever be transformed digitally? Why didn't anyone warn us how hard it would be?
[00:03:31] Airport billboards had it wrong. Transformation's messy, and it's long, and we're failing miserably.
[00:03:42] But wait. Maybe transformation isn't just about technology. Maybe it starts with us, the leaders.
[00:03:52] When we address the questions that have always plagued our teams, like when we ship code to production, how disturbing are those screams? Have our results been feeble because we only pretend to train our people? And do we know our systems are secure? Are we sure?
[00:04:16] And are we helping solve the problems that face our world today? If it's hybrid work or climate change, how much do we change to lead the way? If you want to turn into a butterfly, you've got to break down into goop and die. It's painful and it's gross, but if we try, well, we just might be transformed.
[00:04:40] Digitally doesn't happen at once, but continuously. May not feel it happening, but gradually it will grow. When we flex our wings, we'll achieve maturity digitally. Oh, wow, this time you'll see. We'll be transformed, probably.
[00:05:12] Thank you very much.
[00:05:19] Thank you so much. My name is Forrest Brazeal, and it's great to be here with you this evening at the ChatGPT -- I'm sorry, the DevOps Enterprise Summit.
[00:05:29] In fact, I feel like I need to clarify: every song you'll hear tonight is 100% organic, AI-free, human-generated content. Just have to say that. Bard can't do this yet. ChatGPT isn't to blame for this, at least not yet.
[00:05:42] Anyway, I work at Google Cloud. I'm the head of developer media, which is a wonderful way for me to help educate people about the cloud in fun and unusual ways. But before that, I did cloud and DevOps kinds of things at a whole bunch of different companies.
[00:05:57] And this is what would happen to us. We would go on these multi-year DevOps transformation odysseys, and at the end of that time, sometimes it seemed like all we would have accomplished is refactoring ourselves into slightly different silos.
[00:06:12] But can I confess something to you tonight? Can I be real with you? Come on, you need to give back. I can't do this all by myself.
[00:06:20] I kind of like silos. I really do. I like them. They're warm, they're comfortable, they're safe. They're great at optimizing collaboration within functional groups. Sometimes they're full of corn. There's really no downside to a silo until it's time to change the status quo, time to change the way we do things, and then they become missile silos.
[00:06:41] I had seen this so much at different places in my career that I felt like I had to write a song about it. And so I did. The song is called "The Reorg Rag." Would you like to hear it? Good. All right, let's do it.
[00:06:52] Just like most reorgs, this song is a little too difficult for me to pull off. So good luck to us all.
01"The Reorg Rag"
[00:07:10] [singing] Well, our company used to have these big silos. We had dev and ops in two different rows, and they'd rarely communicate except to fight.
[00:07:19] So we said we're gonna build ourselves a dedicated DevOps team because everybody loves a go-between. We're gonna break the silos down. We're gonna do it right.
[00:07:28] We're gonna do a little re-org rag. Yes sir, this time we're gonna fix this company. We're gonna have this DevOps thing in the bag after one more round of the re-org rag.
[00:07:44] Well, the DevOps team started hot as heck, but they soon became kind of a bottleneck, and instead of two silos, now we had three.
[00:07:53] So we said, we're gonna make dev do their own ops. Now it's all in the cloud; they can figure out how. And we believe the term for this is SRE.
[00:08:02] We're gonna do another re-org rag. Yes siree, this time we're gonna fix this company. We're gonna storm the silos and capture the flag after one more round of the re-org rag.
[00:08:16] Well, it turns out ops is incredibly tough, and our devs soon said they had enough. So we brought back ops, but now they needed more. They said, you're gonna need a platform team to guard and guide, and the DBAs wouldn't come along for the ride. We had silos everywhere, and they were all at war.
[00:08:35] So here's where we're at. I'm on the tooling team, which is owned by dev, but functionally, it rolls up under this new Cloud CoE, which was recently spun out of CorpIT with a dotted line to Security. And some of those folks still report to me. I think technically I'm my own VP.
[00:08:53] We're gonna need a bigger rag. Yes sir, somebody put us out of our misery, because my optimism may start to sag if we play another round, another play, another round, another play, another play, another play, another round of the re-org rag.
[00:09:31] Thank you very much. But this is what we do. No matter how many organizational structures or strictures we adopt, we tend to fall back into these adversarial org patterns, because fundamentally, we're human. We're a tribal species.
[00:09:46] We like to form groups around things, whether it's a technological thing or an organizational fad. And then once we've formed this new tribe, this new group, what do we do? We dig a deep moat around ourselves, and we take up pitchforks and torches to ward off the bad, ignorant people who are still using the paradigm from like four minutes ago.
[00:10:08] We do this, I think, to disguise how small and insecure we really feel. Speaking of things that are small and insecure: who here is using serverless functions in production?
[00:10:21] Some of you are. What about Kubernetes? Who's using Kubernetes in production today? There's got to be more of you than that. We just had like 10,000 people here in Amsterdam for KubeCon. Who's on the fence, thinks this is all just hype at this point? I know you're out there.
[00:10:38] Okay, listen. We're gonna settle this tonight once and for all, and we're gonna do it in the time-honored fashion. We're gonna do it with a serverless versus containers rap battle.
[00:10:48] Now, as you've noticed, I'm one person, so I cannot do this by myself. I'm gonna need your help. This half of the room, my right, your left, I need you to be my serverless fans. When I have my serverless hat on, I need you to show serverless some love.
[00:11:05] And then this side of the room, when I have my containers hat on, I need you to show containers some love. Let's practice. Containers, let me hear you. Got a robust containers fan club. Serverless, what do you have to say? Okay, this is gonna be close.
[00:11:25] Are we ready? Drop the beat.
02Serverless vs. Containers Rap Battle
[00:11:31] [as serverless] You can call me crazy, but the way my mama raised me, servers make me nervous, but serverless never faze me. Maybe I'm just lazy, but why deploy a box when abstraction brings the action for a fraction of the cost? My code is in a zip file, requirements in the pip file. Cram in a function, and bam, all I do is ship while standing proud in the cloud, on the shoulders of the giants. What you need a server for? The value's in the clients.
[00:11:55] [as containers] But I wish there was something more portable to wrap my app in. Oh wait, there is: it's called containers. They make crap happen. Check it out. I can build the way I'm used to, open source dope, of course, run where I choose to. Automate and orchestrate with Kubernetes at the helm, hand on my tiller, I'm a killer in the service realm. Plus all these hello-world tutorials are slick, so someone call the Docker because the stack is looking sick.
[00:12:16] [as serverless] See, this is what you do. It's insane. You overcomplicate the name Kubernetes, so arcane you spell it with an eight. I'm no cloud economist, but I'm sure you don't want to miss the savings you could find if you put your mind to simplifying this. Are you building Pixar? Is your name John Lasseter? Then why you need a service mesh? Why you need Ambassador, Istio, Miss me, yo. This kingdom isn't magic. You throw it in production and the outage gonna be tragic.
[00:12:34] [as containers] Hey, take a breath, man. You ran kind of long there. Functions have a timeout; you gotta save your song there. You've got limitations; I run applications. Every enterprise has their eyes on containerization. Why long jobs? Real time? Doesn't matter. I eat hard problems for mealtime, so pass the platter. So many industries left to disrupt. I know your cold start makes it hard, but try to keep up.
[00:12:54] [as serverless] Check it: one, two, three. He's getting warmer. Wait and see, the latencies are worse than your complacency. Face and see, you're basically chasing a place you don't want to be. My services improve all by themselves; they get better. Meanwhile, you're out of luck stuck checking out the cheddar. Remember Spectre and Meltdown? You were up all night. Me, I slept. That's right. The cloud provider kept it tight. You can patch your runtimes; I'll have happy fun times delivering value while, pal, you fight the same old fight.
[00:13:14] [as containers] That's great, but wait. Let's use our brains here. Yo, I got constraints here. I'm running Java 8 here, digging in the brownfield, moving the ball downfield. Can't rearchitect it all until we look respectable. I just want to build more. That's what I get billed for. Functions are a thing our predecessors would've killed for. I know. I'm just saying we're in a different state of being. Yeah, but functions are amazing. Wait, are we agreeing?
[00:13:36] [singing] Obviously both of us have the same destination: get rid of heavy lifting without differentiation. So whether your abstraction is a function or a node, you can get a lot of traction on the old cloud road. If your app goes down at 3:00 a.m., and it will, you gotta own that. It's your problem still. There's no silver bullet, hocus-pocus magic guarantee, but when business is your focus, you'll be where you want to be.
[00:14:05] Thank you very much. But let's get serious now for a minute.
[00:14:10] Serverless isn't the problem, is it? Containers aren't the problem. There's no magic reorg that's gonna deliver value somehow to our customers. Even the right vision isn't enough.
[00:14:22] We can attend all these talks and buy every one of IT Revolution's wonderful books. You can probably ask ChatGPT to generate a 12-point business turnaround plan, and I'm sure it will do it for you. But none of it means anything without other people, who have to execute these things.
[00:14:42] You can't transform your organization until you empower the people in it to transform themselves.
[00:14:46] I have one more song to play for you tonight, but before I do, I want to introduce you to a young man named Daniel Singletary.
03Daniel Singletary and the Cloud Resume Challenge
[00:15:06] This is Daniel. I met Daniel about three years ago. At that time he was a residential and commercial plumber working in Atlanta, Georgia, doing all the things you might expect a plumber to do: long days, on his feet a lot.
[00:15:21] Shortly before I met him, Daniel got called out to a shopping mall there in Atlanta because the owners were reporting a strange and horrible smell permeating the entire building. Daniel heads out there, and right away the smell hits him in the face. First troubleshooting step, he heads into the bathroom, takes one of the toilets off the floor, and immediately he's hit with a blast of noxious air.
[00:15:45] The way he described it later was like having a leaf blower blowing sewer gas in your face. Daniel assures me that not only is that not something that should happen, it shouldn't even be possible. Sewer pipes don't blow air at you.
[00:16:02] How do you break down a problem like that? In Daniel's case, he came back to the business with a colleague. Over the next two days, they started at opposite ends of the building and systematically worked their way toward each other as they unscrewed, tested, and re-screwed every pipe fitting in the building. They could not find where this air was escaping.
[00:16:21] Finally, they narrowed the problem down to two businesses in the center of the complex, a restaurant and a nail salon. They knew they had to try something else. This is where they tried a smoke test.
[00:16:31] Who here has done a smoke test in the context of software engineering? Quite a lot. Did you know that's a real term with a real smoke-related meaning? In plumbing, what a smoke test is, is you get up on the roof, find a vent, and drop a smoke bomb down it. Anywhere smoke can get out, the noxious gas could be getting out as well, but we can see smoke with a flashlight.
[00:16:54] That clears up the problem in pretty short order. It turns out that someone has tied the vent hood for the restaurant's commercial stove into the sewer system. It was around that time, I believe, that Daniel decided he was ready to look for a new career, possibly one where he could choose what to smell from time to time.
[00:17:16] That is how Daniel and I met, because his roommate worked in IT. His roommate introduced him to something that I had created called the Cloud Resume Challenge, a community initiative designed to help people like Daniel who are looking to break into technology.
[00:17:29] If you've not seen the Cloud Resume Challenge, you can check it out at cloudresumechallenge.dev. It looks like a wall of text. It's not a video course you watch. It's not a certification you can take, although we do encourage you to get a certification. It's a set of steps, basically a spec, that helps non-traditional career changer learners -- people without a traditional four-year computer science degree -- take their first steps toward a career in cloud and get learning and building like an engineer.
[00:17:58] This is a real architecture diagram created by one of our challengers. They're putting their personal website in the cloud. They've got to use infrastructure as code; there's a small serverless API; they've got a full CI/CD pipeline; they're programming in at least two programming languages; there's authentication, networking, and security involved.
[00:18:20] If you can complete all 16 steps of the Cloud Resume Challenge -- and many people don't -- you've done something not only that a lot of four-year university computer science graduates haven't done, but a lot of established professionals haven't either. Every day we have people three or four years into their career taking this on and saying, yeah, I've never strung all those skills together. I'd like that on my resume.
[00:18:38] So put yourself in Daniel's shoes and imagine how he felt looking at this architecture diagram for the first time, thinking, is this what I have to do to get a career in technology? I'll tell you exactly what was going through his mind. He said, that looks familiar.
[00:18:56] So he went out and bought a whiteboard. This is his real whiteboard. He sent me a picture of it, and he called what he drew there to make sense of the project an engineered print, because that's what they call it in the trades. He did not realize he was drawing his very first cloud architecture diagram.
[00:19:11] Over the next couple of weeks, Daniel spent part of his time working on sewer pipelines and the rest working on CI/CD pipelines. He was working in the back of his truck during the day. Then he'd get home at the end of these 11-hour days and he's slinging Python and JavaScript. He's learning about YAML for the first time. Let's pour one out for this man.
[00:19:32] He thinks about giving up, I'm sure, multiple times. But at the end of that time, he's actually done it front to back. He's built this stack, he's deployed it in the cloud, and it is live today. You can check it out at dsresume.com.
[00:19:42] Is it the most beautiful CSS-formatted website you've ever seen? No, but it doesn't matter. I call it the greatest resume I've ever seen because look at the credentials Daniel has on his resume. He's a licensed journeyman plumber. He's certified in backflow prevention and cross-connection control inspection, and he's got four certifications on AWS. How cool is that?
[00:20:06] The last step of the Cloud Resume Challenge is you've got to write a blog post telling the world about what you did. Daniel's post was called "A Plumber's Guide to Cloud." It went viral on LinkedIn and was eventually viewed more than 200,000 times. A couple of months later, Daniel started his new career as a DevOps engineer at a large enterprise.
[00:20:28] When I talked to the hiring team that had brought Daniel on, they told me they liked that he had done this technical project. It was cool to see he had some aptitude for that, but let's be real: he's still very inexperienced. He's gonna need a lot of training. But what they really liked was that story of the leaf-blower sewer.
[00:20:46] Think about what's being demonstrated by that story. Daniel's able to take a gigantic, pretty diffuse problem -- this entire building stinks -- and break it down into its constituent parts. He's able to work with a colleague, basically pair program, to divide and conquer, make their way toward the solution as they eliminate possibilities. He gets tracing and observability; that's what that smoke testing thing was. And perhaps most importantly, he has a great grasp of business continuity.
[00:21:13] He knew that he couldn't take apart the entire shopping mall's plumbing system for an undefined length of time while he pieced through all the parts. That wasn't gonna work. He was gonna have to troubleshoot it live. And so he did.
[00:21:25] For that reason and many others, Daniel was a rock-solid real-world operator. He's killing it in his career still today, as are Jacob, who was an infectious disease researcher; Brooke, who worked for T-Mobile; Sha Diamond, who was in HR; and many other wonderful people whose stories you can read at cloudresumechallenge.dev. Many of these folks are now my friends and colleagues in the industry.
[00:21:55] I wish that I had so many more of those stories to tell you. But unfortunately, a lot of times what I hear six, nine months later is these Cloud Resume champions who've completed the project come back to me and they say, it's not that we're failing interviews. We just can't get job interviews.
[00:22:13] We're being told, oh, we don't have any positions open for junior engineers right now. Or we're being told, oh yeah, we have this new-grad program, so just go be 22 years old again and come fresh out of college and then we've got a perfect spot available for you. All these things that we tell new grads, and it's not working for them. It's not really working for us either.
[00:22:40] We're depriving ourselves of the opportunity to have some of these incredible real-world operators as part of our teams. I get it. We all have priorities and budget constraints and hiring freezes. But at some point, we're gonna have to think about this more strategically.
[00:23:01] We've been hearing so much about generative AI today. We heard about Jevons paradox a little earlier, this idea that as you make something more efficient to consume, and there's elastic demand for it, that demand actually goes up. We're seeing that now with code. The more of this code we generate, the more rock-solid real-world operators we're going to need to maintain that code and harden it and be on call for it at three in the morning.
[00:23:27] These people have to come from somewhere. Where are they all gonna come from? How many are we gonna need? I'll give you a number: 40 million. This is how many new cloud builders Google Cloud is saying we're going to need in the industry by the end of the decade. AWS and others have put broadly similar numbers out there. There is no university computer science pipeline big enough to force all those people through it.
[00:23:49] We have to think bigger. We have to be more creative with where we look for this talent and how we create programs to nurture and onboard it inside of our teams.
[00:24:01] So what does that actually look like? Standing up here on a rock-solid, practical level, what does that look like? The team that brought Daniel Singletary on actually went on, because they had such a good experience with Daniel, to hire I think six more non-traditional career changers onto that team within that enterprise. In order to do that, they kind of had to back their way into creating what we would probably call an apprenticeship program.
[00:24:30] That's fascinating because guess who had lots of experience being in an apprenticeship program? It was Daniel. That's how he got into plumbing. A lot of the things he had to do to be a successful plumbing apprentice translated right along to tech.
[00:24:46] At least in the US, when you have an apprenticeship in the trades, typically you don't pin onto one practitioner; you get rotated around to multiple plumbers or electricians or whatever through your local trade board. In the same way, when Daniel got to tech, his team had to identify some specific senior engineers who were willing basically to pair program with him and help him get up to speed.
[00:25:16] Those people in turn had to be incentivized. This isn't something you can necessarily just ask people to take on if they're not bought into it. You had to make sure that was part of a growth ladder for those seniors just as much as it was for Daniel.
[00:25:29] At the same time, you had to get top-down support from leadership as well. You've gotta have buy-in on this. In the plumbing world, apprentices get paid. They don't get paid much, but they do get paid for every hour that they worked. In the same way, if you're gonna bring someone in, you've got to set expectations with your leadership in tech that this person might not be 100% ready to contribute at an autonomous level yet, but we believe there's value in investing in this person. We think there's a sustainable talent pipeline to be developed by doing this kind of thing.
[00:25:59] Finally, there had to be some structure and objective goals around that program. When Daniel completed his plumbing apprenticeship, he knew he was done because he was able to pass the test, get certified as a journeyman plumber, and start out on his own.
[00:26:12] What does that look like in tech? Does it look like reaching a point where we feel we can take the associate off your title? Does it look like being able to be on call all by yourself without somebody holding your hand? Does it look like completing some number of certifications, or whatever we've identified as the benchmark for being able to take the kid gloves off and let you have access to production? Whatever that is for your team, you need to know it going in so you don't leave people in this state of limbo.
[00:26:41] We're all asking for help with different things. My ask is to help me help you help others. We've had thousands, probably tens of thousands at this point, of people go through the Cloud Resume Challenge over the past three years. These people have wonderfully unique stories of how they've been brought up in tech, the challenges they've faced trying to get hired, what has worked for them. We need to be sharing those practices more broadly.
[00:27:05] Please come find me afterwards or reach out to me. You can see my Twitter and my email address here. This is not part of my regular job. It's just something that I do because it needs to be done, and I would love your help in doing it.
[00:27:19] Let me leave you with this. Every one of us here in this room is here because at some point in the past, somebody took a chance on you. Somebody looked at you when you had no experience and said, that person has the aptitude and motivation to succeed in this field, and I believe I can teach them the rest. And you proved them right. So how are you paying that forward?
[00:27:45] I have one more song to play for you tonight. I actually wrote this song for my daughter, who just celebrated her first birthday. And as you go back to your teams and create cultures of mentorship and belonging there -- wasn't that smooth, from mic to mic? -- I hope this will resonate with you. This song is called "You Belong."
04"You Belong"
[00:28:18] [singing] You won't always feel like a rockstar. Sometimes you'll struggle just to carry the tune. In the mirror, you'll see an impostor, and you'll wonder if the whole world's smarter than you and stronger and faster, and you are a disaster. But baby, you are wrong.
[00:28:43] Because you belong, even when you don't believe it. You belong. I've got help here when you need it. Just hold on. Yeah, the road ahead is steep, so take my hand. We'll make the leap together. Oh, oh, you belong.
[00:29:07] Sometimes you'll feel kind of burned out. You'll meet some folks who seem diffuse and unfair. Or you'll have a dream, and it turns out that no one else around you seems to care. And when you see what you're up against, it's fine to fight, to feel incensed, or just to move on.
[00:29:28] Because you belong in a place where you're supported. You belong where you're challenged and rewarded. Just hold on. As you feel your power growing, you can choose the way you're going. And it's oh, oh, where you belong.
[00:29:50] One of these days you're gonna reach the mountaintop. Yeah, it's not if, but when. And when you do, I hope you'll take a beat to stop and celebrate how far you've come, remember where you've been, and then reach back to those behind you. Shine a light so they can find you. Help the ones you can, then sing to them again.
[00:30:14] Tell them you belong when you're new and nervous. Then you still belong. I know it 'cause I've been you. Just hold on. Yeah, the road ahead is steep, so take my hand. We'll make the leap together. Oh, oh, you belong. Let's sing it together now. Oh, oh, you belong. We can do better than that. Let's try it again. Oh, oh, where you belong.
[00:31:05] Thank you very much. Enjoy the rest of DevOps Enterprise Summit.
Host Transition (Gene Kim)
[00:31:20] Yeah. I'm not crying. You're crying. All right, everybody. Now some of you were probably wondering what this piano was doing up on the stage this day.