Transformed! A Musical DevOps Journey with Forrest Brazeal
Transformed! A Musical DevOps Journey with Forrest Brazeal
Developer Communities, Google
All right. So to describe, uh, the next talk, I mean, know exactly the moment in 2021 when Richard Retter, whose work I've admired for many, many years, sent something to me on Twitter. It was this amazing song that the brilliant and talented Forrest Braze had composed and performed. Uh, I was just leaving a parking lot of a grocery store with all three of my kids. I pulled over because I loved it so much, and I played it for my kids, and they loved it just as much as I did. So obviously, I then tried to find everything that Forest has created, and in my conversation with them, I had learned that we had many things in common, including an obsession with anything related to Hamilton, Linman, Wallman, Miranda, and so forth. And I must admit, in my grandest and wildest dreams, if there ever were to be a Phoenix Project musical, I'm so hoping it would be something that forest, uh, creates. But the other thing I learned is that this is only just one small sliver of the amazing things that Forest has done. In particular, I continue to be blown away by the successes that from an initiative he created called the Cloud Resume Challenge. So I asked for if he'd be willing to talk about this amazing work and share with us some of his other amazing creations. And I am so delighted that he said yes. Here's Forest.
Good evening. It's 5:41 PM Do you know where your, uh, DevOps transformation is? <laugh>, we know where it started, don't we? It always starts the same way. It starts with high hopes. It starts with big dreams. It starts with a vision.
Picture us glorious rising up at last to be
Cloud. Big data and AI will bid our legacy ways. Goodbye. We're caterpillars now, but we're gonna fly. Yeah, we're gonna be transformed digitally. Never done it before, but how hard could it be? All these slide decks look so great. Set. The deadline won't be long to wait till we declare victory. Digitally leave. How hard could it be? Years pass, alas, fighting still. The fires we fought before our POCs looked great. So why are we not seeing r o I and all the time our spend is climbing high? Or will we ever be transformed digitally? Why didn't anyone warn? That's how hard it would be. Airport billboards had it wrong. Transformations messy, and it's long, and we're failing miserably. But wait,
Maybe transformation isn't just about technology. Maybe it starts
With us. The leaders, when we address the questions that have always plagued our teams, like when we ship code to production, what is causing all those screams? Have our results been feeble? Because we only pretend to train our people. And do we know our systems are secure? Are we sure? And are we helping solve the problems that face our world today? If it's hybrid work or climate change, how must we change to lead the way? If you want to turn into a butterfly, you've got to break down into goop and die. It's painful and it's gross, but if we try, well, we just might be transformed. Digitally doesn't happen at once, but continuously, we may not feel it happening, but gradually we'll grow and flex our wings. We'll achieve maturity digitally. Oh, wow. This time you'll see we'll be transformed probably.
Thank you very much. My name is Forrest Braze. I work for Google Cloud. I'm their head of developer media, which basically means I get to make all kinds of extremely unusual, uh, content that developers will enjoy. Uh, but before I worked at Google, I did cloud and DevOps kinds of things for a whole bunch of different companies. And this is what would happen to us. We would go on these multi-year DevOps transformation Odyssey. And at the end of that time, it sometimes felt like all we would've accomplished would've been to have refactored ourselves into slightly different silos. Uh, but can, can I be real with you? Can I confess something to you tonight? Yeah. Okay, good. Good. Yeah. You're gonna have to give back tonight. Uh, I can't do this all by myself. Uh, so listen, I kind of like silos. I do. I no, I, listen.
Hear me out. I like them. They're safe, they're comfortable. Uh, they're great at optimizing communication within functional groups. Sometimes they're full of corn. Uh, there's really no downside to a silo until it's time to change the status quo, right? Time to change the way we do things. Uh, and then they become missile silos. And I had seen this so much at, at different places over the course of my career. Not Google. I'm not talking about Google. Don't come after me Google. But I'd seen this at, at so many places over the course of my career, uh, that I felt like I had to write a song about it. And the song is called The Reorg Rag. Would you like to hear it? Yes. All right. So just like most reorgs, uh, this song is a little bit too difficult for me to pull off accurately in real time. Uh, so good luck to us all. Well, our company used to have these big silos. We had Devon Ops in two different rows, and they'd rarely communicate except to fight. So we said we're gonna build ourselves a dedicated DevOps team because everybody loves a go-between. We're gonna break these silos down. We're gonna do it right. We're gonna do a little rejo yesy this time. We're gonna fix this company. We're gonna have this DevOps thing in the bag after one more round of the reorg rag.
The DevOps team started hot as heck, but they soon became kind of a bottleneck. And instead of two silos, now we have three. So we said we're gonna make Dev do their own ops. Now it's all in the cloud. They can figure out how, and we believe the term for this is ss r e. We're gonna do another reorg. Yes, re this time we're gonna fix this company. We're gonna storm the silos and capture the flag after one more round of the reorg rag.
It turns out Ops is incredibly tough. And Rd dev soon said they'd had enough. So we brought back ops, but now they needed more. They said, you're gonna need a platform team to guard and guide, and the DBAs wouldn't come along for the ride away at Silos everywhere. And they were all at war. So here's where we're at. I'm on the tooling team, which is owned by Dev, but functionally, it rolls up under this new cloud, c o e, which was recently spun outta corp it with a dotted line to security. And some of those folks still report to me. I think technically I'm my own vp. We're gonna need a bigger reorg. Yes, siree, somebody put a sound of our misery, because my
Optimism may start to s if we play another round, other play another round, other the play, another play, another play, another play, another play, another play, another play, another play, another round, other reorg rag.
Thank you so much. But this is what we do, isn't it? And, and I think no matter what organizational structures or, or strictures we adopt, we're always at risk of falling back into these kind of adversarial, uh, org patterns. Because fundamentally, we're human. We're, we're a tribal species. We like to to in-group around things, whether it's, uh, an organizational thing, whether it's a technology fad, we do that for sure. And then once we form this, this new group, what do we do? We, we dig a deep mode around ourselves, uh, and we take a pitchforks and torches to, to ward off the bad ignorant people who are still using the paradigm from like four minutes ago. Uh, and, and we do that to, to distract ourselves from how small and insecure we really feel. Uh, speaking of things that are small and insecure, who here is using serverless functions, uh, in production?
Anybody? Yeah. Some of you. Great. How about, uh, Kubernetes? Who's using Kubernetes in production today? Yeah, some of you are wonderful. Uh, who, who thinks it's all just a bunch of hype at this point? <laugh>, pathetic fence sitters. I see where you are. Well, listen, we are gonna settle this tonight once and for all. We're gonna do it in the time honored fashion. We're gonna do it with a serverless versus containers wrap battle. How are you in <laugh>? Alright, let's do it. Okay. Now listen, as you've noticed, I'm one person, so I cannot do this by myself. I'm going to need your help. So this whole side of the room, uh, everybody down here, whoever's up in the balcony, right down the middle, you're gonna be my serverless fans for tonight. Sorry if that hurts you. You're gonna be serverless fans for tonight. And when I have my serverless hat on, can you see that when I have my serverless hat on?
I need you to make some noise. I you to show some love. Let me hear all my serverless people. All right, all good. Okay. All right. We've got a robust fan club. Now, on the other hand, when I have my containers hat on, I need this whole half of the room to show me some love. So when containers gives out a sick burn, right? I need y'all to go, Ooh, lemme hear y'all do that. Oh, oh, all right. Okay. I don't dunno. I, I don't know if any of us can win. It'll be all right. We'll, we'll get through it. Uh, okay. Wonderful. Well, in that case, I think we're ready to go. Are we ready? Woo. Okay. Drop the beat.
Yeah, you can call me crazy. By the time my mama raised me, servers make me nervous, but serverless never phase me. Maybe I'm just lazy. But why deploy a box? When abstraction brings the action for refraction of the cost, my code is in a zip file requirements in a PIP file, cram it in a function, and bam, all I do is ship while standing proud in the cloud, on the shoulders of the giants. What? You need a server for the values and the clients? There we go. Yeah, but I wish there was something more portable to wrap my app in. Oh wait, there is, it's called containers. They make crap happen. Check that out. I can build the way I'm used to. Open source dope, of course, run it where I choose to automate and orchestrate with Kubernetes at the helm hand on my tiller.
I'm a killer in the service realm. Plus all these Heller world tutorials are slick. So someone called the docker 'cause the stack is looking sick. See, this is what you do. It's insane. You overcomplicate the name Kubernetes so arcane. You spell it with an eight. I'm no cloud economist. But I'm sure you don't wanna miss the savings you could find if you put your mind to simplifying. This are your building. Pixar is your name John Lasseter. Then why you need a service mesh? Why You need Ambassador Istio, miss eo, this is kingdom isn't magic. You throw it in production and the outage gonna be tragic. Hey, take a breath, man. You ran kinda long there. Functions have a timeout. You gotta save your song there. You got limitations. I run applications. Every enterprise has their eyes on containerizations. Why long jobs? Real time doesn't matter.
I eat hard problems for mealtime. So pass the platter. So many industries left to disrupt. I know your cold start makes it hard, but try to keep up. Check it. 1, 2, 3. He's getting warmer. Wait and see the latency's no worse than your complacency. Face it. See, you're basically chasing a place you don't wanna be. My services improve all by themselves. They get better. Meanwhile, you're outta luck. Stuck, chucking out the cheddar. Hey, remember Specter and meltdown? You were up all night. Me, I slept. That's right. The cloud provider kept it tight. You can patch your runtimes. I'll have happy, fun times delivering value while pal you fight the same old fight. That's great. But wait, let's use our brains here. Yo, I got constraints here. I'm running Java eight here, digging in the brownfield, moving the ball. Downfield can't re-architect it all until we look respectable. I just want to build more. That's what I get. Build for. Functions are a blast. Our past cells would've killed for, I know, I'm just saying we're in a different state of being. Yeah, but functions are amazing. Wait, are we agreeing? Obviously both of us have the same destination. Get rid of heavy lifting with that differentiation. So whether you're abstraction is a function or a node, you can get a lot of traction on the old cloud road if your app goes down at 3:00 AM and it will, you gotta own that. It's your problem still. There's no silver bullet hocus pocus magic guarantee. But when business is your focus, you'll be where you want to be.
Thank you very much.
Alright, well I have one more song to play for you tonight, but before I get there, I wanna introduce you to a fellow named Daniel Singletary. Because let's be real, serverless isn't the problem. Containers aren't the problem, right? There's no magic reorg that's gonna deliver value to our customers. Even the right vision isn't enough. We can attend all these talks. We can underline all the right parts of the Phoenix project. None of it means anything without what the people who have to execute these things can't transform your organization unless you empower the people in it to transform themselves. Alright? So I wanna introduce you to Daniel Singletary. This is Daniel. I met Daniel a couple of years ago, right at the beginning of the pandemic. At the time he was a residential and commercial plumber working in metro Atlanta, doing all the things you would expect a plumber to do on his feet a lot long days.
Shortly before I met him, he got called out to a shopping mall there in Atlanta because the owners were reporting a strange and horrible smell, pervading the entire building, maybe something you would expect a plumber to deal with. So Daniel heads out there right away, the smell hits him in the face. He can, he can, uh, detect it. So he heads into one of the bathrooms. First troubleshooting step, takes one of the toilets off the floor. Instantly he's hit with a blast of noxious air. Now, I'm not a plumber, but Daniel assures me that not only is this not something that should happen, it shouldn't even be possible. You shouldn't have currents of air blowing through your, your sewer system. So what, how do you break down a problem like that? Well, in Daniel's case, he came back to the business with a colleague and they started at opposite ends of the shopping mall.
And they started working their way systematically toward each other. Over the next couple of days, they systematically unscrewed tested and re-scored every pipe fitting in the building trying to figure out where this horrible smell could be escaping. They couldn't find it. Finally, they got down to two businesses in the center, a restaurant and a hair salon, and they had to try something else. So this is where they tried a smoke test. Who here has done a smoke test in the context of software engineering? Yeah, a lot of you have. Did you know that's a real term with a real smoke related meaning? I didn't until I met Daniel. Guess what a smoke test is in the context of plumbing. You get up on the roof, you find a vent, and you drop a smoke bomb down it. That's a smoke test in plumbing. Uh, and the reason they do that is, you know, anywhere that this gas could be getting out, smoke can get out too.
But the difference is we can see smoke with a flashlight. So that eventually clears up the problem. Turns out that someone has tied the vent hood for the restaurant's commercial stove into the sewer system. And that's what's, it's always something like that, isn't it? That's what's causing this, this error. Uh, it was around that time I believe that Daniel decided that he was ready to look for a new career <laugh>, maybe one more, where he could choose what to smell from time to time. Uh, and that is how Daniel and I met because his roommate worked in it, and his roommate pointed him to something that I had created called the Cloud Resume Challenge. If you haven't seen the cloud resume challenge, you can check it out. It's at cloud resume challenge.dev. And it looks like this, it looks like a wall of text. It's basically a spec that tries to get people coming from non-traditional backgrounds.
So people without a like four year college degree in computer science or whatever, uh, people who haven't worked with cloud before, it tries to get them hands-on and learning and building like an engineer. So it's not a video course you watch, uh, it's not, you know, a certification you take, although we do encourage you to get a certification as you can see. But basically, uh, it's giving you a spec with some instructions to put your personal website in the cloud. Uh, but there's specific things you have to do, you've got to use. Well, here I'll show you, uh, an example. This is a, oh, there we go. This is a recent challengers architecture diagram. And you can see they're having to use source control. There's some infrastructure as code code built in here. They're using Terraform. Uh, they've got a cloud storage bucket going on.
They built a small serverless, a p i. Uh, and if you can complete all this, which is not a given, many people try this and can't get through it, you've actually built something pretty impressive. You've built something not only that a lot of people can't do, coming out of a university computer science program, you built something that a lot of professional cloud engineers haven't done. In fact, every day we have people working on the cloud resume challenge. Who are, they've, they're three, five years into their career and they say, oh yeah, I've never strung all those skills together at once. I would definitely like to try that. So put yourself in Daniel's shoes and imagine how he felt looking at these kind of things for the first time, trying to decide if he wanted a career in technology. What do you think his reaction would've been?
I'll tell you exactly what his reaction was. He said, that looks familiar. So he went out and he bought a whiteboard. This is his real whiteboard. He sent me a picture of it. And this is what he drew to try to make sense of the project he was about to undertake. Daniel called what he was drawing and engineered print because that's what they call it in the trades. He did not realize he was drawing his very first cloud architecture diagram. So over the next few weeks, Daniel spent part of his time working on sewer pipelines and the rest of the time working on C I C D pipelines. Uh, again, a real picture, Daniel, let's not ask what's on his shirt.
So Daniel writes a blog post about his experiences. It's actually the first requirement for, or I should say the last requirement for any one of these projects is you have to write a blog post about it and share it. He called his blog A Plumber's Guide to Cloud. It ended up going viral on LinkedIn. It was viewed more than 200,000 times. And just a month or two later, Daniel started his new career as a DevOps engineer at a large enterprise. And as well, he should, right? Because when I talked to the hiring team that had brought Daniel on, they told me they, they liked that he had done some technical things. I mean, that was cool, but let's be real, he's still very green, right? He is got a lot to learn. But they really liked that story of the leaf blower sewer. Think about what's being demonstrated by that story.
Daniel is able to take a gigantic kind of diffuse problem. This entire building stinks and he's able to break it down into achievable steps. He's able to work with a colleague basically to pair a program, break the problem down, eliminate possibilities step by step until they arrive at the only solution he gets tracing, he knows how to use tracing techniques. That's what that smoke testing thing was. And most importantly, this man has a great grasp of business continuity, right? He knew that he couldn't just kick out everybody in the restaurant while he meticulously took apart their entire plumbing system. That wasn't gonna work. He was gonna have to figure out how to troubleshoot this live. And so for that reason, and many others, he was a fantastic fit on his team and continues to kill it in his career today. In fact, that team, uh, at that enterprise has actually gone on to hire, I think, six more non-traditional career changers, uh, because of the success that they had with Daniel.
And Daniel's not the only person who's got a success story outta the cloud resume challenge. Uh, there's Jacob who was a, um, infectious disease researcher. Uh, he's got some great covid pandemic stories that are applied to tech. Uh, there's, uh, Chi Diamond who was in hr. There's Brooke who was a worked at T-Mobile. Uh, there, we've had loggers go through. We've had professional poker players go through it. And you can read a lot of their firstname.lastname@example.org. Uh, many of these people are now my friends and colleagues in the industry. But I wish I had even more of those stories to tell you because let's be real. You know, it's tough to get a job in this industry at the best of times. It's tough to break in. Um, I think a lot of us came up in an era where you could kind of just say, well, I scraped and scrambled around for a few years and eventually I turned out okay.
So the next generation turned out okay too. I don't think that was ever true. It's definitely not true. Now, Google Cloud is saying that we're gonna need something like 40 million more cloud builders in the industry by the end of this decade. A w s and others have put broadly similar numbers out there. Those people are not all going to come through that traditional four year golden path, uh, where they, they get a computer science degree and, and take advantage of some of the like green bean and new grad programs that you may have at your companies. They're not just gonna be floating around on the open job market. We have to think bigger. We have to get more creative with how we source and find this talent. And we have to be more creative and more intentional about how we onboard and create experiences for those people.
It's interesting, you know, this company that brought Daniel on, I told you they brought on several more people, uh, from the Cloud resume challenge. And what they ended up doing was kind of backing their way into creating an apprenticeship program at their company as opposed to a traditional developer onboarding process. And I think that's interesting because <laugh> guess who had lots of experience going through an apprenticeship program, it was Daniel, right? That's how he got into plumbing, you know, very familiar with that. And some of the things that he had to do in the plumbing apprenticeship world, like going out and working under different plumbers. I dunno if you know this, but in the trades, you don't just pin onto one, uh, practitioner. You, you get rotated around to different people through your local trade board. Um, some of those people were probably better to work with than others, but he had to do that.
And in the same way when Daniel came onto his, his team, uh, in tech, uh, they had to find senior engineers who were willing to basically pair with him, him. Um, and they had to set expectations with those people. And it wasn't all their senior engineers. They found some people that were willing to do this. Uh, and then they kind of grew from there. Uh, in the same way, you know, Daniel got paid in his plumbing apprenticeship. He, he didn't do it pro bono. He didn't get paid much, but he did get paid for every hour that he worked. So this company that brought Daniel on had to figure out a way to make sure that he got paid a living wage. They had to set that expectation with, uh, their leadership team that hey, even though this person isn't going to be, you know, contributing at full capacity for a little while, we think that there is substantial value in investing in someone like this.
We think there's value in, uh, making sure that they have what they need to succeed. And then finally, Daniel, in his plumbing apprentice world, when he completed a year or two, he was able to go on and get licensed. He was able to start out as a journeyman plumber on his own. So there had to be some clear progression and goal set as he was going through that process. It wasn't just nebulous and open-ended. Uh, and that's what ended up having to happen in tech as well. There had to be this clear understanding in the organization of this person is an apprentice. You have to have different expectations for them right now, uh, but they will get to a point where they're able to help. And here's what that's gonna look like and I want to talk to you much more about that, uh, later tonight.
So my ask to you, everybody's been asking for help with different things. My ask is for you to help me, help you help others. I want to talk to you after this about maybe apprenticeship programs you might be running at your company. I want to learn from you. I wanna share what I've learned. We've put thousands of people through the cloud resume challenge now over the past two and a half years. Um, they all have unique stories. They have unique stories of, of where they've landed and how they've gotten, uh, brought up in the industry. And we need to be sharing those practices more broadly. So please come find me. You can find me. Uh, you can see cloud resume challenge.dev there, you see my Twitter, you see my email address? Please reach out. I really wanna hear from you. This is not part of my regular job, it's just something I do, uh, because it needs to be done and I really want your help in doing it.
Let me leave you with this bottom line. Wait, wait, lemme leave you this bottom line. Every one of us here in this industry, every one of us sitting in this room tonight got here because at some point in the past, somebody took a chance on you. Somebody looked at you when you had zero experience and said, that person has the aptitude and motivation to succeed in this field. And I think I can teach them the rest. And you prove them right? So how are you paying that forward? I have one more song to play for you tonight. I actually wrote this song a few months ago for my, uh, daughter who was born in May.
So like her hello, lost this audio.
I wrote this song for my daughter, uh, who was born in May. So like her. This song is new. Uh, you're the first public audience actually ever to hear it. And I hope that as you go back and work with your teams, uh, mentor them, create cultures of belonging there, that this will resonate with you. Now, the song is called You Belong.
You won't always feel like a rock star. Sometimes you'll struggle just to carry the tune in the mirror. You'll see an impostor and you'll wonder if the world smarter than you and stronger and faster, and you are a disaster. But baby, you're wrong. 'cause you belong. Even when you don't believe it, you belong. I've got help here when you need it. Just hold
Yeah, the road ahead looks steep. So take my hand, we'll make the leap together. Oh, you belong.
Sometimes you'll feel kind of burned out. You'll work with folks who seem diffuse and unfair, or you'll have a dream. And it turns out then no one else around you seems to care. And when you see what you're up against, it's find a fight to feel and sense or just to move on. 'cause you belong in a place where you are supported, you
On where you're challenged and rewarded. Just hold
As you feel your power growing, you can choose the way you're going. And it's oh,
Oh oh where
And one of these days you are gonna reach the mountain top. Yeah, it's not if, but when and when you do, I hope you'll take a beat to stop and celebrate how far you've come, remember where you've been, and then reach back to those behind you. Shine a light so they can find you. Help the ones you can then sing to them again. Tell 'em
When you're new and nervous, then you still belong
I know it 'cause I've been, you just, hold on. I know the road ahead is steep, so take my hand. We'll make the leap together.
Oh, oh, oh, you
Oh, come on. Sing it with me now.
Oh, oh, oh,
That was good. Let's try it again.
Oh, oh, oh.
Where you belong.
Thank you very much. Enjoy the rest of DevOps Enterprise Summit.
Whoa. Hey, hey. I looked up and
It's about that time.
I bet you all have been wondering what that piano was up here for the entire day. Well, now, you know, right? Fors, thank you for the songs and for the stories and for the call to action. Uh, I think we, I think we just witnessed something very, very special, uh, here at the summit today. So thank you so much for that
Outta curiosity. Does anyone want an encore who hereby round applause like an encore? All right, Forrest, you're on? Yes.
It's 6:10 PM Uh, no. So, uh, every year at Black Hat, who here has been to Black Hat? Some of you? Yeah, they do a thing. They're called the Pony Award, spelled P W N I E. Uh, and it's basically their, they get up dressed up and they roll out a red carpet. Um, and it's their way to kind of gently poke fun at some things that happen in the security community during the year. So they give awards like Best Exploit and, uh, lamest Vendor Response. Um, it, it's kind of borderline toxic, but that's black hat for you. Uh, so anyway, they, like any good award show, they also give an award for best song. And uh, last year they gave the pony award for best song to the song I'm about to play for you now, which is, uh, a very sober and fact filled reminder of how to protect yourself against ransomware. Why we're dealing with all this ransomware. It is called the ransomware song. One day
I asked my teacher, what use is math to me? She answered, when you're older, some day, my boy, you'll see there's a world of computer systems out there full of valuable data and not secured with care. And you can make a fortune in ransomware with a little bit of math. It's called encryption. Just a little bit of math. Cause a conniption lock the data, hide the key, they'll pay up eventually. That's the power of math. Ransomware is big now because it's organized. You can buy it as a service and sell it for a prize. There's a whole world of tempting targets to hack governments, hospital schools to attack. And they pay up in Bitcoin, hard to trace back. It's all just math. We call it crypto. Just a little bit of math. No need to tiptoe hackers, whisper. Don't be nervous. We take pride in customer service. Well then I told my teacher, I'm feeling terrified. How can we protect ourselves from all this cyber crime? She said, don't click weird links or use password. 1, 2, 3, make offline backups. Invest in security. And that's when I realized how screwed we might
Do the math. So many systems, if they all took a bath, wouldn't we miss them? Next time there's no meat or water or gas and we slide a little closer to the day of wrath, you can blame it or some Russian sociopath. But personally, I blame math.
If you appreciated what Forrest did up here and you can't do it yourselves, <laugh>, could you all show how much you appreciated him by getting out overseas? And gimme a big round of applause. Thank you so much.