Transformed! A Musical DevOps Journey with Forrest Brazeal
Transformed! A Musical DevOps Journey with Forrest Brazeal
Developer Communities, Google
Alright. Uh, I know exactly the moment in 2021 when, uh, Richard Retter, uh, whose work I've admired for many years, uh, sent me this tweet. Uh, and it was, uh, a tweet about this amazing song that the brilliant and talented forest Brazil had composed and performed. Um, I was just leaving the parking lot, uh, a grocery store with, uh, all three of my kids. I pulled over and played it for them, and they loved it just as much as I did. So obviously, I then tried to find everything that for Brazee had ever created. And in my conversations with them, I discovered that we had a bunch of things in common, including an obsession, uh, with everything related to, uh, Hamilton and Lin win Miranda. And, uh, I must confess in my grandest and wildest dreams, if there ever were to be a Phoenix Project musical, I'm hoping it would be as amazing as what Forest has created. Uh, the other thing I learned about Forest is that, you know, that musicals are just one sliver of the amazing things that Forrest has done. In particular, I continue to be blown away by the success stories from an initiative he created called the Cloud Resume Challenge. So I asked Forrest if he'd be willing to talk about his amazing work and share some of his other amazing creations with us. And I'm so delighted that he said yes. Here's Forest.
Good evening, 5:28 PM Uh, do you know where your digital transformation is? <laugh>. We know where it started, don't we? It always starts the same way. It starts with high hopes. It starts with big dreams, it starts with a vision. Picture us glorious rising up at
Last to be
Cloud. Big data and AI will bid. Our legacy weighs goodbye. We're caterpillars now, but we're gonna fly. Yeah, we're gonna be transformed digitally. Never done it before, but how hard could it be? All these slide decks look so great. Said The deadline won't be long to wait till we declare victory digitally. How hard could it be? Years pass, alas, fighting still. The fires we fought before our POCs looked great. So I are we not seeing r o i and all the time our spend is climbing high? Or will we ever be transformed digitally? Why didn't anyone warn? That's how hard it would be. Airport billboards had it wrong. Transformations messy and it's long and we're failing miserably. But wait,
Maybe transformation isn't just about
Maybe it starts
With us. The leaders,
When we address the questions that have always plagued our teams, like when we ship code to production, how disturbing are those screams? Have our results been feeble? Because we only pretend to train our people? And do we know our systems are secure? Are we sure? And are we helping solve the problems that face our world today? If it's hybrid work or climate change, how must we change to lead the way? If you want to turn into a butterfly, you've got to break down into goop and die. It's painful and it's gross, but if we try, well, we just might be transformed. Digitally doesn't happen at once, but continuously may not feel it happening. But gradually we'll grow when Flex our wings will achieve maturity
Oh wow. This
Time you'll see
Beach runs fall, probably.
Thank you very much.
Thank you so much. My name is Forrest Bra Brazil, and it's great to be here with you this evening at the chat. G p t I'm sorry, the DevOps Enterprise Summit. Uh, in fact, I feel like I need to clarify. Every song you'll hear tonight is 100% organic ai free human generated content. Uh, just have to say that. Uh, Bard can't do this yet. Uh, chat. G p t can't isn't to blame for this, at least not yet. Uh, anyway, I, I work at, uh, Google Cloud. I'm their head of Developer Media, uh, which is a wonderful way for me to help educate people about the cloud in, in fun and unusual ways. Uh, but before that, I did cloud and DevOps kinds of things at a whole bunch of different companies. And this is what would happen to us. We would go on these multi-year DevOps transformation Odyssey. And at the end of that time, sometimes it seemed like all we would've accomplished, uh, is refactoring ourselves into slightly different silos. But, um, can I, can I confess something to you tonight? Can I be real with you? Okay. Come on. You need to get back. I I can't do this all by myself. Yeah. All right. Good. Uh, I kind of like silos. I really do. I like them. They're, they're warm, they're comfortable, they're safe. Uh, they're great at optimizing collaboration within functional groups.
Uh, sometimes they're full of corn. There's just,
No downside to a silo until it's time to change the status quo, time to change the way we do things. Uh, and then they become missile silos. And I had seen this so much at different places in my career that I, I felt like I had to write a song about it. And so I did. And the song is called The Reorg Rag. Would you like to hear it?
Yes. Okay, good.
All right. Let's do it.
Just like most reorgs, uh, this song is a little too difficult for me to pull off. So good luck to us all.
Our company used to have these big silos. We had Devon Ops in two different rows, and they'd rarely communicate except to fight. So we said we're gonna build ourselves a dedicated DevOps team because everybody loves to go between. We're gonna break these silos down. We're gonna do it right. We're gonna do a little read yes Siry this time. We're gonna fix this company. We're gonna have this DevOps thing in the bag after one more round of the reorg rag. Well, the DevOps team started
Hot as heck, but they soon became kind of a
Bottleneck. And instead of two silos, now we have three. So we said we're gonna make Dev do their own ops. Now it's all in the cloud. They can figure out how, when we believe of the term Boer, this is Ss r e, we're gonna do another re Yo. Yes, re it is time. We're gonna fix this company. We're gonna storm the silos and capture the flag after one more round of the reorg. Well, it turns out Ops is incredibly tough, and our dev soon said they had enough. So we brought back ops, but now they needed more. They said, you're gonna need a platform team to guard and guide, and the DBAs wouldn't come along for the ride. And we had silos everywhere, and they were all at war. So here's
Where we're at. I'm on the tooling team, which is owned by Dev, but
Functionally it rolls up under this new cloud, c o e, which was recently spun out a corp it with a dotted line to security. And some of those folks still report to me. I think technically I'm my own vp. We're a bigger, yes, somebody put us out our misery because my optimism may start to sag if we play another round of the play, another round of the play, another play, another play, another play, another play, another play, play another round of the reorg rag.
Thank you very much. Uh, but this is what we do. You know, and, and I think, uh, no matter how many organizational structures or or strictures we adopt, we tend to fall back into these kind of adversarial org patterns because fundamentally, we're human. We're a tribal species. We like to form groups around things, uh, whether it's a, a technological thing and organizational fad, right? We do that for sure. And then once we formed this new tribe, this new group, what do we do? We, we dig a deep mode around ourselves, and we take up pitchforks and torches to ward off the bad ignorant people who are still using the paradigm from like four minutes ago. And, uh, we do this, I think to, you know, disguise how small and insecure we really feel. Uh, speaking of things that are small and insecure, who here is using serverless functions in production?
Some of you are. Yeah, that's great. Uh, what about Kubernetes? Who's, who's using Kubernetes in production today? Anybody? Oh, come on. There's gotta be more of you than that. We just had like 10,000 people here in Amsterdam for coup con. Yeah, a few of you are, uh, who's on the fence? Thinks it's all just hype at this point. Yeah. Uh, I, I know you're out there. Uh, okay, well listen, we're gonna settle this tonight once and for all. And we're gonna do it in the time honored fashion. We're gonna do it with a serverless versus containers wrap battle. All right? So now as you, as you've noticed, I'm one person, so I cannot do this by myself. I'm gonna need your help. So this whole half of the room, my right, your left, I'm gonna need you to be my serverless fans for tonight, okay? So when I have my serverless hat on, I need you to make some noise. I need you to show serverless some love. All right? And then this side of the room on my left, when I have my containers hat on, I need you to show containers some love. Okay? So let's practice containers. Let me hear you, let me hear you. Uh, all right. Okay. Got a robust containers fan club serverless. What do you have to say?
Okay. I don't know why this is gonna be close. We'll see who triumphs. Alright, are we ready? Okay, drop the beat. There we go. Yeah, you can call me crazy. By the time my mama raised me, servers make me nervous, but serverless never phased me.
Maybe I'm just lazy.
But why deploy a box when abstraction brings the action for a fraction of the cost, my code is in a zip file requirements in the PIP
File, cram in a function, and bam, all I do is ship while standing proud in the cloud, on the shoulders of the giants. What you need a server
For the values and the clients. Yeah. Yeah. But I wish there was something more portable to wrap my app in. Oh wait, there is, it's called containers. They make crap happen. Check it out. I can build the way I'm used to open source dopa course, run it where I choose to automate and orchestrate with Kubernetes at the helm hand on my tiller Ramma killer in the service realm. Plus all these HelloWorld tutorials are slick. So someone called the docker 'cause the stack is looking sick. See, this is what you do. It's insane. You overcomplicate the name Kubernetes so arcane. You spell it with an eight. I'm no cloud economist, but I'm sure you don't wanna miss the savings you could find if you put your mind to simplifying this. Are you building Pixar? Is your name John Lasseter? Then why you need a service mesh?
Why you need ambassador is d o miss me. This kingdom isn't magic. You throw it in production and the outage gonna be tragic. Hey, take a breath, man. You ran kinda long there. Functions have a timeout. You gotta save your song there. You've got limitations. I run applications. Every enterprise has their eyes on containerizations. Why long jobs? Realtime doesn't matter. I eat heart problems from mealtime, surpass the platter. So many industries left to disrupt. I know your cold start makes it hard, but try to keep up. Check it. 1, 2, 3. He's getting warmer. Wait and see the latency's no worse than your complacency face. And see, you're basically chasing a place you don't wanna be. My services improve all by themselves. They get better. Meanwhile, you're outta luck stuck checking out the cheddar. Hey, remember specter and meltdown? You were up all night. Me, I slept.
That's right. The cloud provider kept it tight. You can patch your runtimes. I'll have happy, fun times delivering value while pal you fight the same old fight. That's great. But wait, let's use our brains here. Yo, I got constraints here. I'm running Java eight here, digging in the brownfield, moving the ball. Downfield can't re-architect it all until we look respectable. I just want to build more. That's what I get. Build for. Functions are a blast. Our past those would've killed for, I know, I'm just saying we're in a different state of being. Yeah, but functions are amazing. And wait, are we agreeing? Obviously both of us have the same destination. Get rid of heavy lifting without differentiation. So whether your abstraction is a function or a node, you can get a lot of traction on the old cloud road if your app goes down at 3:00 AM and it will, you gotta own that. It's your problem still. There's no silver bullet hocus pocus magic guarantee. But when business is your focus, you'll be where you want to be.
Thank you very much. But, uh, let's get serious now for a minute. Serverless isn't the problem is it? Containers aren't the problem. There's no magic reorg that's gonna deliver value somehow to our customers. Even the right vision isn't enough. We can attend all these talks and by every one of it, revolution's, wonderful books. Uh, you can probably, uh, ask chat g p t to generate a, you know, 12 point business turnaround plan. And, and I'm sure it'll do it for you, but none of it means anything without what other people, right? Who have to execute these things. You can't transform your organization until you empower the people in it to transform themselves. I have one more song to play for you tonight, but before I do, I wanna introduce you to a young man named Daniel Singletary. Let's see if I can figure this out.
This is a pointer. Well, let's see, I'm not known. There we go. Solved one problem today. Alright, this is Daniel. Uh, Daniel I met about three years ago. Uh, at that time he was a residential and commercial plumber working in Atlanta, Georgia. And, uh, he was doing all the things you might expect a plumber to do long days on his feet a lot. Shortly before I met him, Daniel got called out to a shopping mall there in Atlanta because the owners were reporting a strange and horrible smell permeating the entire building, maybe something you'd expect a plumber to deal with. So Daniel heads out there, and, uh, right away the smell hits him in the face. It's inescapable. So first troubleshooting step adds into the bathroom, takes one of the toilets off the floor, and immediately he sit with a blast obnoxious air. The way he described it later was, he said it was like having a leaf blower, blowing sewer gas in your face.
Now, I'm not a plumber, but Daniel assures me that not only is that not something that that should happen, it shouldn't even be possible, right? Sewer pipes don't blow air at you. So something really weird is going on here. How, how do you go about breaking down a problem like that? Well, in Daniel's case, he came back to the business with a colleague, and over the next two days, they started at opposite ends of the building and systematically worked their way toward each other, uh, as they unscrewed tested and re-scored every pipe fitting in the building, and they could not find where this air was escaping. Finally, they narrowed the problem down to two businesses in the center of the complex, a restaurant and a nail salon. And they knew they had to try something else. So this is where they tried a smoke test.
Who here, uh, has done a smoke test in the context of software engineering? Yeah, a lot. Almost all of us have, right? Did you know that's a real term with a real smoke related meaning? I didn't. But, uh, turns out that in plumbing, what a smoke test is, is you get up on the roof and you find a vent and you drop a smoke bomb down it. That's a smoke test in plumbing. And you do that because anywhere that smoke can get out, while the, the noxious gas could be getting out as well, but we can see smoke with a flashlight. So that clears up the problem in pretty short order. It turns out that someone has tied the vent hood for the restaurant's commercial stove into the sewer system. And, uh, yeah, that's, uh, that's, uh, explains that it was around that time I believe that, uh, Daniel decided that he was ready to look for a new career, possibly even one where, where he could choose what to smell from time to time.
Uh, and that is how Daniel and I met because his roommate worked in it. His roommate had introduced him to something that I had created called the Cloud Resume Challenge, which is a community initiative designed to help people like Daniel who are looking to break into technology. If you've not seen the Cloud resume challenge, you can check it out. It firstname.lastname@example.org. And it looks like this, it looks like a wall of text. Uh, it's not a video course you watch, it's not a certification you can take, although we do encourage you to get a certification as you can see. But it's a set of steps, basically a spec, uh, that helps non-traditional career changer learners. So think people without like a, a traditional four year college degree in computer science, take their first steps toward a career in cloud and get some learning and building like an engineer.
Uh, and I'll show you what that looks like. So this is a real architecture diagram created by one of our challengers. Uh, they're putting their personal website in the cloud and look at all the things that they've got going on here. Uh, they've got to use infrastructure as code that's required in the spec. They've got to, uh, there's a small serverless, a p i happening. Uh, they're, they've got a full C I C D pipeline. They're programming in at least two programming languages. Uh, there's authentication, networking, security involved. If you can complete all 16 steps of the cloud resume challenge and, and many people don't, uh, you've done something. Not only that a lot of, let's say four year university computer science graduates have done, but a lot of established professionals haven't either. Every day we have, you know, people that are three or four years into their career who are taking this on saying, yeah, I've never strung all those skills together.
I'd like that on my resume. So put yourself in Daniel's shoes and imagine how he felt looking at this architecture diagram for the first time, thinking, is this what I have to do to get a career in technology? What do you think was going through his mind? I'll tell you exactly what was going through his mind. He said, that looks familiar. So he went out and bought a whiteboard. This is his real whiteboard. He sent me a picture of it and he called what he drew there to make sense of the project and engineered print, because that's what they call it in the trades. He did not realize he was drawing his very first cloud architecture diagram. Over the next couple of weeks, Daniel spent part of his time working on sewer pipelines and the rest working on C I C D pipelines. Uh, he was working in the back of his truck during the day, then he'd get home at the end of these 11 hour days.
Daniel's post was called a Plumber's Guide to Cloud. It went viral on LinkedIn was eventually viewed more than 200,000 times. And just a couple of months later, Daniel started his new career, uh, as a DevOps engineer at a large enterprise. And as well, he should, right? Because when I talked to the hiring team that had brought Daniel on, they told me they liked that he had done this technical project. It was cool to see he had some aptitude for that, but let's be real, he's still very inexperienced. He's gonna need a lot of training. Uh, but what they really liked was that story of the leaf blower sewer. Just think for a moment about what's being demonstrated by that story. Okay? Daniel's able to take a gigantic, pretty diffused problem. This entire building stinks, break it down into its constituent parts. He's able to work with a colleague, basically to pair program, to divide and conquer, make their way toward the solution as they eliminate possibilities.
He gets tracing and observability. That's what that smoke testing thing was, right? And perhaps most importantly, he has a great grasp of business continuity. He knew that he couldn't take apart the entire shopping mall's plumbing system for an undefined length of time while he pieced through all the parts that wasn't gonna work. He was gonna have to troubleshoot it live. And so he did. And for that reason, and many others, uh, Daniel was a rock solid, real world operator. And, uh, he's killing it in his career still today, as are Jacob, who was an infectious disease researcher. He's got some great covid pandemic stories as they apply to tech. Uh, and Brooke who worked for T-Mobile and Shadi who was in hr and many, many other wonderful people whose stories you can email@example.com. Many of these folks are now my friends and colleagues in the industry.
And I wish that I had so many more of those stories to tell you. But unfortunately, a lot of times what I hear six, nine months later is these cloud resume champions who've completed the project come back to me and they say, it's not that we're failing interviews, we just can't get job interviews. We're being told, uh, we don't have any positions open for junior engineers right now, or we're being told, oh yeah, we have this, this new grad program. So just go be 22 years old again and come fresh outta college and then we've got a perfect spot available for you. Right? All all these things that, that we tell new grads and, uh, it's, it's not working for them. Uh, and it's not really working for us either. We are depriving ourselves of the opportunity to have some of these incredible real world operators as part of our teams.
And look, I mean, I get it. We all have priorities and budget constraints and hiring freezes and who knows what all going on. That's true of any micro moment in tech. But at some point, we're gonna have to think about this more strategically, aren't we? I mean, we've been hearing so much about generative AI today. We heard about Jevons paradox a little earlier, this idea that as you make something more efficient to consume, uh, and there's elastic demand for it, that demand actually goes up. You know, we're seeing that now with code, uh, <laugh>. The more of this code we generate, the more rock solid real world operators, we are going to need to maintain that code and harden it and be on call for it at three in the morning, right? And, uh, these people have to come from somewhere. Where are they all gonna come from?
How many are we gonna need? I'll give you a number. That number is 40 million. This is, uh, how many new cloud builders Google Cloud is saying we're going to need in the industry by the end of the decade. A W Ss and others have put broadly similar numbers out there. There is no university computer science pipeline big enough to force all those people through it. We have to think bigger. We have to be more creative with where we look for this talent and how we create programs to nurture and onboard it inside of our teams.
So what does that actually look like? That's easy to say. Standing up here on a rock solid, practical level, what does that look like? You know, it's interesting. The team that brought Daniel Singletary on, they actually went on because they had such a good experience with Daniel. They went on to hire, I think six more non-traditional career changers onto that team within that enterprise. And in order to do that, they kind of had to back their way into creating what we would probably call an apprenticeship program. And I think that's fascinating because guess who had lots of experience being in an apprenticeship program? It was Daniel. That's how he got into plumbing, right? And a lot of the things he had to do to be a successful plumbing apprentice translated right along to tech. I'll give you three quick examples. So at least in the US I'm not sure about Europe.
Uh, when you have an internship in the trades, typically you don't pin onto one practitioner, you get rotated around to multiple plumbers or electricians or whatever through your local trade board. And, uh, I'm sure some of those people were better to work with than others. I'm not claiming that, you know, the trades are this, this panacea panacea of empathy and, and nurturing behavior. Uh, but you know, in the same way, when Daniel got to tech, his team had to identify some specific senior engineers who were willing basically to pair program with him and, and help him get up to speed. And that means those people in turn had to be incentivized, right? This isn't something you can necessarily just ask people to take on if they're not bought into it. You had to make sure that that was part of a, uh, growth ladder for those seniors just as much as it was for Daniel.
Uh, at the same time, you had to get top-down support from leadership as well. You've gotta have buy-in on this. I mean, in the plumbing world, you know, an intern, uh, apprentices get paid. Uh, they don't get paid much, but they do get paid for every hour that they worked. And the same way, if you're gonna bring someone in, you've gotta set expectations with your leadership in tech that hey, this person might not be 100% ready to contribute, you know, at a, at an autonomous level yet. But we believe that there's value in investing in this person. We think that there's a sustainable talent pipeline to be developed by developed by doing this kind of thing. Uh, and then finally, there had to be some structure and some objective goals around that program, right? Uh, you know, when Daniel had completed his plumbing apprenticeship, he knew he was done because he was able to go pass the test and, uh, get certified as a journeyman plumber and start out on his own.
What does that look like in tech? Does it look like, you know, hey, you're gonna reach a point where we feel we can take the associate off your title? Does it look like you're gonna be able to, uh, be on call all by yourself without somebody holding your hand? Does it look like you have to complete this number of certifications or whatever we've identified as a benchmark of, you know, being able to kind of take the kid gloves off and, and let you have access to production, whatever that is for your team. You need to know what going in so you don't leave people in this state of limbo. And I would love to talk to you much more about that later tonight. We're all asking for help with different things. My ask is to help me, help you help others. Okay? We've had thousands, probably tens of thousands at this point of people that have gone through the cloud resume challenge over the past three years.
These people have wonderfully unique stories of how they've, uh, been brought up in tech, the challenges they've faced trying to get hired, what has worked for them. We need to be sharing those practices more broadly. So please come find me afterwards or reach out to me. You can see my Twitter, my email address here. I would love to talk to you. Um, this is not part of my regular job. It's just something that I do because it needs to be done. And I would love your help in doing it. Lemme leave you with this. Every one of us here in this room is here because at some point in the past, somebody took a chance on you. Somebody looked at you when you had no experience and said, that person has the aptitude and motivation to succeed in this field. And I believe I can teach them the rest. And you prove them right? So how are you paying that forward? I have one more song to play for you tonight. I actually wrote this song from my daughter who just celebrated her first birthday. Uh, and as you go back to your teams and create cultures of mentorship and belonging there, hope. Oh, it wasn't that smooth. Look at that. Went from Mike to Mike. Great work. Uh, I hope that as you go back to your teams, uh, that this will resonate with you. This song is called You Belong.
You won't always feel like a rockstar. Sometimes you'll struggle just to carry the tomb in the mirror. You'll see an imposter and you'll wonder if the whole world smarter than you and stronger and faster, and you are a disaster. But baby, you are wrong
Belong. Even when you don't believe it, you belong. I've got help here when you need it. Just hold on. Yeah, the road ahead. Look steep. So take my hand, we'll make the leap together. Oh, whoa. Oh, oh, you belong. Oh, sometimes you'll feel kind of burned out. You'll meet some folks who seem diffuse and unfair, or you'll have a dream. And it turns out that no one else around you seems to care. And when you see what you're up against, it's find to fight, to feel and send store just to move on. 'cause you belong in a place where you're supported, you belong, where you're challenged and rewarded. Just hold on. As you feel your power grow, you can choose the way you're going. And it's
Oh, whoa, oh,
Whoa, where you belong.
And one of these days you're gonna reach the mountaintop. Yeah. It's not if, but when and when you do, I hope you'll take the beat to stop and celebrate how far you've come. Remember where you've been, and then reach back to those behind you. Shine a light so they can find you. Help the ones you can.
Sing to them again. Tell 'em you belong. When you're new and nervous, then you still
I know it. 'cause I've been, you just hold
Yeah, the road ahead is steep. So take my hand. We'll make the leave together.
Oh, oh, oh, oh.
You belong. Oh,
Oh. Let's sing it together now.
Oh, oh, oh, oh.
You belong. Oh,
Oh. We can do better than that. Let's try again.
Oh, oh, oh, oh.
Where you belong.
Oh, oh, oh.
Thank you very much. Enjoy the rest of DevOps Enterprise
Yeah. I'm not crying. You're crying. <laugh>. Alright everybody. Now some of you were probably wondering what this piano was doing up on the stage. This.