00:00:01

In my opening remarks. I mentioned Dr. Richard Cook from the safety culture community in a previous conference, he told us something amazing. He said there are certain types of stories that you'll never be able to hear on stage. Instead, you hear them after the sessions, most likely at the bar, after there have been a few drinks. And that's when the real lessons are told, great practice comes from experience. He says, and experience comes from bad practice. This struck me as profoundly important. The only people who got to hear those stories have to be in the right time, the right place and hanging out with the right people. So we wanted to bring those stories so that you can hear them as well. Thus, we created the DevOps confession format. We've collected stories that we've anonymized and we'll share with you. So please welcome. One of our programming committee members, Cornelia Davis, CTO of Weaveworks, who will be reading one of these stories over to you, Cornelia.

00:00:59

Hello, my name is Cornelia Davis and I am part of the programming committee for the DevOps enterprise summit. And today I'm pleased to bring you this DevOps confession. We call this one cert apocalypse several years ago, after many years of operational struggles and an inability to improve our operational environment. We realigned development in level two operations into cross-functional build run teams. That is our developers would be taking on some of the operational responsibilities. The existing operations leader did can continue providing some operational support, including level one help desk and other Itel processes such as change and incident management. There were many objections and there was a lot of background chatter to this plan, including DevOps will not work here. Engineers don't understand ops and engineers should never access production. After the dust settled from the reorg, we set out to improve operations by treating the operational processes as engineering problems.

00:02:10

We began automating a lot of things. This included a heavy focus on infrastructure as code and using version control for all scripts and tooling, that supported operations. We started looking for easy wins to reduce toil on the teams who were in constant firefighting mode, working around the clock to keep up. We found that we had hundreds of certificates to renew each month, renewing a single cert required a ticket. A team would process that ticket and return a certificate after which we would manually apply it to production. Lead times were very long and manually retrieving the cert and properly updating it. And production was error, prone and dangerous often causing an outage. Fortunately, our certificate store had an API and we felt that this was a great opportunity to automate a tedious and error prone process. We had some of our best engineers design and implement a process that would check the certificate store.

00:03:14

And if a cert was up for renewal, it would revoke and delete the old certificate and issue a new one to be installed into production. As script was built that would iterate through all certs and do the deletion and issue process. During development, the engineer went about testing this for the first time and this script access the store to carry out the process. Unfortunately, this script was pointed at the one and only production instance of the certificate store. The developer watched in horror as the script began deleting and revoking every certificate in the enterprise. Fortunately, the developer carefully watching the output scroll by reacted quickly telling the script when it had only invalidated a handful of certificates, had the developer not noticed it would have revoked and recreated every certificate in the enterprise, including those that support our communication systems and email the developer quickly escalated to the enterprise in. Okay, I'm going to start that paragraph again and see what I can do to add it. Fortunately, the developer care,

00:04:45

Fortunately, the developer carefully watching the output scroll by reacted quickly, killing the script. When it had only invalidated a handful of certificates, had the developer not noticed it would have revoked and recreated every certificate in the enterprise, including those that support our communication systems and email, the developer quickly escalated to management and the team worked to restore the invalidated certificates, minimizing the impact of the mistake. Following the incident we did of course put proper controls into place to segment the production certificate store. From that of the lower environments, we have never shared this story outside of a select few people at the time, the organization was still raw from the reorg and would have looked to punish the individual and potentially unwind our move to DevOps.

00:05:44

Well, hello everybody. My name is Damon Edwards. John Willis. You might know us from the, uh, DevOps cafe, uh, podcast, which, uh, often features a lot of, uh, folks from the greater DevOps enterprise summit community. So we're happy to be hosting the lightning talk session. Uh, if you want to catch more of a podcast, you can go over to DevOps, cafe dot Oregon register to be notified of the upcoming next generation of the podcast. Uh, but today Don John, we're here to introduce some lightning talks. Uh, can you get, tell us all of the, what lightning talks are all about?

00:06:17

Yeah. For those you don't know the lightning talk format, it's basically a 20 slides, uh, 15 seconds a slide and five minutes. What's interesting is usually it's very stressful live virtually. We've asked people to create that same stress level by automatically advancing. And I don't think we had any cheaters. So I think we're pretty good here.

00:06:36

Yeah. Yeah. It's a, it's a virtual high-wire act for the performers and, uh, it's a little Intermezzo or a pallet cleanser for, uh, for everybody else. So, uh, hopefully you enjoy them. Uh, John, who do we have? Uh, whose first,

00:06:48

First we have cats with tell and J bloom and I think it's going to be interesting. Um, there talks around social technical and actually Kat has actually worked with James. So it'll be interesting to see both of them tell their stories.

00:07:02

Yeah, I'm excited. So, uh, here we go. The first two lightning talks of the virtual DevOps enterprise summit.

00:07:18

Hi, I'm Kat Patel. And today I'm going to be telling you an alternative dev ops origin story. I will also warn you that at any moment, my son could come busting through that door and I probably won't even stop the recording. So let's go back to the beginning. In the beginning, there was agile, there weren't any women or were there, I don't know. It depends on who created the guests list. Am I right? Hashtag not my uncle in the beginning, there was agile. It started with developers, but it became very clear very quickly that they would need to go out and conquer other disciplines within software development, product development. So we see the testers, the BAS project managers, all sorts of folks join the agile tool. Why? Because we need to prioritize obtaining new information from the quickly changing market. So we need all of these disciplines to work together, to enable those 10 X developers to get that information from the market as quickly as possible.

00:08:23

So we have to support the other disciplines. Well, what do you do after you have conquered all the other disciplines and software development and product development, the next frontier is going to be the deployment and the operations of that software. So agile of course decides, now we need to further expand our agile empire and we need to conquer ops. Not really that difficult. You just add some words to the agile manifesto. It's a little bit weird because now we're doing both development and production. Uh, so we got to come up with a new name. That's understood. That's how DevOps is born, momentous occasion for us all. And I think that's the history that we all take for granted. Now I'm going to challenge you to consider a little bit of a different take. What if dev ops is not an extension of the agile empire?

00:09:23

What if dev ops is a reaction to agile? And to illustrate my point here, I will start by asking a question. Why do companies die? Uh, if we're going to listen to our fine friends at the Santa Fe Institute, we can think about companies as an organism, a complex system that is finely balanced between the energy that it takes to metabolize new information and the energy that it takes to keep up the maintenance that keeps the company alive, keep the lights on. So for us in working in companies, we have to balance the metabolism of new information coming from the markets so that we can stay alive. We can continue to meet the needs of the customers. We have to balance that with keeping the lights on with maintaining our systems. How do we as technologists, metabolize new information, which working software with code, right?

00:10:25

Of course. So if we want to get lots of information, we've got to get whole lots of code. And then what happens to all that code? Well, my friends, we have to maintain it. And that's where this balance becomes extremely difficult to, um, maintain. So what I guess what I'm challenging you to consider is what if dev ops is not an extension of the, the optimized for metabolism, mindset of agile? What if it's a new mindset that challenges us to consider the maintenance costs and the balance that we have to strike. We just have this one set of resources of energy at time to invest in both the metabolism of new information and the maintenance of what is already existing. So if we have to invest a lot in maintenance, we won't have very much leftover for metabolism. And I think this is really clear if we look at some of the principles that we hold dear in DevOps, they're all about being informed about the system and making really informed tradeoffs.

00:11:35

And I know it sounds like I'm given the agile folks a really hard time with this more and more and more mentality, but we do need that in order to keep up with the market, we do need to prioritize the metabolism of new information. However, we also need to be really mindful of that balance and we need to invest in minimizing our maintenance debt so that we can maximize our investment in metabolizing new information from the market. And so I don't think dev ops came from agile. I think it is a reaction to agile. And that's my talk. Thank you.

00:12:18

Good evening, everybody. I'm Nate Ashford, and this is my story. This is a story of how I changed the world, starting with my world. I am a lot of things, but basically I like to make stuff that works. And I like to celebrate it with the awesome people who do it with me. And sometimes we don't get to celebrate very often. And that's where transformation comes in. The first few times I was involved in a transformation. I was filled with this eager idealism, that things are going to be so much better and everyone is going to win. I do still believe in the winning and the better, but it can get a little bit messy and I can't always save everyone, but I can help someone. And if I can make a difference in the life of one person, that can be enough. And there's only one person I can actually change me.

00:13:15

Two years ago, I was a mess. I was not having the impact I wanted to have. Despite working long hours, I had a vision for change, but it wasn't going anywhere. I was spinning my wheels and the stress was eating me up and I was eating to match. I peaked at 255 pounds. I loved martial arts, but I couldn't ever go to class because I was constantly injured. I was in pain. I was exhausted. I was depressed. I even hid behind the plant and the family photo, something had to change, but conveniently, I fancy myself a change agent. I help people change. I'm a people. So I took myself on as a client and I started with what I know, start with why I want to live long enough to see my grandkids and to see them grow up and maybe even see their kids core values.

00:14:11

I decided that I like vegetables. Make the, make the work visible. I started tracking my calories, my water intake, my exercise, even my sleep measure, what matters. I monitored the outcomes that I wanted in weight, body, fat muscle mass. I set a goal to lose a pound or two a week. I started with just taking a walk half hour at a time than an hour. Then longer as the weight started to come off, I started to mix in some running and then more running. I started working fewer hours and putting more time into self-care like a daily routine in morning. It was I planning and meditation in the evening, reflection and journal writing. I experimented adding and removing things from my routine. As they figured out what works. And a year later I had lost a third of my body weight, 85 pounds. I was happy.

00:15:10

I was loving my work. And after 13 years I finally got my black belt. Life was good. Although I did have a couple of things I wanted to talk to my doctor about, I put off the appointment until the day after my black belt test. So he couldn't tell me not to do it. And we did a simple little blood test, chronic myelogenous leukemia. Those three words changed my life instantly and permanently, but had I not begun my transformation when I did, I would not have caught the signs and I might not have found it in time. The questions in my head will I live? Yes, thanks to powerful drugs taken every day from here on out. Will I lose everything I've worked for for that? I need another principle, error, budgets, big ones. I can't do karate right now. And I can't run as often as I'd like to other things slip as well.

00:16:09

I don't always have my a game and that's okay. Will I live with purpose? This is my most recent one. And I choose, yes. I choose a life that is about more than me managing my symptoms and my side-effects today. I'm about to celebrate my one-year cancerversary on Saturday. I do not have it all figured out, but my error budgets are smaller. I keep trying pressing forward and occasionally panicking and pulling back. But I've learned this. My cancer is not a liability. It is an asset. I'm a better person and a better coach because of it. It has taught me about myself, about empathy, about struggle, about limitations, about letting go. And today I have more impact to create change in the lives of the people around me than I ever had in my previous life, because I have changed and transformation begins with me.

00:17:19

Pretty amazing. Start to the DevOps enterprise summit this year. Don't you think? All right. We're about to head to a break, but I want to share some things to know before you go. Like I said, we're about to break now. And then the breakout talks, we'll start again at 11:25 AM. London time. We have networking time and networking opportunities starting at 2:25 PM. London time. That's birds of a feather, lean coffee and chat roulette. Visit our sponsors of the expo and check out the games and be back here for more keynote talks starting at 4 0 5 London time. All right. Have fun. Learn lots.