00:00:07

Hi there. Thanks for joining us today. We have Tom lemon jelly from stack overflow, and he's going to talk to you today about low context dev ops, Tom, over to you.

00:00:18

Hi, welcome to low context DevOps. My name is Tom Lemoncelli. I'm the manager of the Esri team here at stack overflow. I've been a system administrator for much too long. I blog I tweet and I've written a number of books. And if you believe the back cover of the books, I'm a internationally recognized system administrator and dev ops pundit. Um, I mentioned I work at stack overflow. We have over a hundred million users, uh, in 2019. Uh, the demographers say that there are about 30 million software developers in the world, which means if for every three developers you meet 10 of them use our website. Now what's less known about our website is, uh, stack overflow for teams. Uh, this is a product that gives a private Q and a experience for your team. And maybe we'll talk about that in the, in the future. So my talk has three parts.

00:01:16

Uh, we're going to talk about the sociology concept of high and low context cultures. We're going to talk about low context DevOps, a phrase that I coined recently, and, uh, we're going to end by talking about leadership. So part one high and low context cultures, let's begin with a story starting. Number one, a man is traveling in a foreign land and he's enters a village. He goes into the first shop and the shopkeeper won't talk to him. He goes into another shop and they won't talk to him either. And he's, he's quite perplexed. So he's sitting on the side of the road, wondering what to do. And a little boy comes up to him and says, Mr. The reason that they're not talking to you is first. When you come to a village, you have to talk to the elders and get their blessing. And then once you have their blessing, everyone will talk to you.

00:02:07

Well, he does that and everything goes fine. After that, everyone will talk to him. Well, that night he's thinking to himself, how the heck was I supposed to know that? And amazingly enough, at that same time, the little boy was thinking, how could anyone not know that story? Number two, sorry. Number two is about my first week at stack overflow. I've been at stack overflow for about seven years. And I still remember my training because there was this point where I asked how to create a virtual machine. Now we use a product called VMware and my mentor walked me through the process. It was five very complicated steps and it wasn't written down. Uh, it was just verbally, you know, passed on from one system into the other. And I said, wow, this is really complicated. How, how can anyone, you know, memorize this? And he said, well, we kind of expected anyone that would get through our interview process would be able to just know how to do this kind of stuff.

00:03:16

Well, that night I remember thinking how could I have been expected to know all of that? And, uh, I wonder if he was sitting at home that night thinking, gosh, you know, we've hired this world recognized system administrator. How could he have not known how to do these things? Well, there's a postscript to this story. The next day I got a phone call from my boss who said, uh, we had made a mistake in the process. And that just goes to show that even though, you know, one person who was very experienced, even he had not, you know, memorize the whole process. So starting number two. So these first two stories are examples of high context cultures in a high context, culture communication is implicit things aren't really written down. It's people just know the collective history. People have to read between the lines to understand what's going on.

00:04:11

Often, often what's not said is more important that what is said in a high context culture, uh, we usually rely on long-term relationships. So this, uh, might work better for, you know, uh, situations where you're, you're going to be there a long time. So some examples include like a party with friends or family gatherings. There's just norms and customs that they're not written down, just people know to do them contrast this with a low context culture and a low context. Culture communication is explicit. There are rules. You're told what the rules are and you follow the rules. Knowledge tends to be codified. You know, it's written down, it's public, it's accessible. Um, they make it accessible so that you can follow the rules. A low context, culture works best for short duration, interpersonal connections. So for example, a large airport, you're not going to be at the airport very long.

00:05:10

There's a lot of rules. And that's why they're on signs, signs that you can read, um, and, and learn that way. Another benefit of low context culture is that the knowledge is more transferable because it's written down. The next person can come along and, and read it. Sociologists that created this concept, uh, have ranked languages around the world as being a high context, low context, or somewhere in between. And while it's, this is a generalization, uh, Eastern languages are, tend to be known as being more high context. Western languages tend to be more low context. One of my favorite examples locally of a high context, uh, environment is New York, Penn. So stack overflow is headquartered in New York city. And I live in New Jersey, which means I take the train in to work every day. And Penn station is just an incredibly high context environment. In fact, this is a map of Penn station. This map only makes sense to people that created the map. This is otherwise pretty much useless. You can't even tell where the second floor and the first floor would overlap onto each other. But my favorite thing in Penn station is this sign right here. The sign that says seventh avenue subway to the left. This sign only makes sense. If you knew that 30 years ago, the seventh avenue subway line changed its name to the 1, 2, 3. That is to me, the epitome of a high context culture. Now,

00:06:51

Recently I was thinking about the sign and I realized it's eliminated. There's a light bulb behind it. And the light bulbs don't last 30 years, which means for the last 30 years, someone's been changing that light bulb. And that person is not empowered to tell their boss, Hey, maybe we should update the signs. Talk about low context dev ops. I assert that dev ops environments should strive to be low context. And I hope you agree. You should spend more time working and less time frustrated with roadblocks and information gaps. But most of all, we shouldn't just change the light bulb. We should fix the darn sign. So let's talk about three ways to lower the context of your environment. Carefully constructed defaults making right easy and ubiquitous documentation Carefully constructed defaults means the default should be the way you expect most people to work. So for example, When you joined a new company as an engineer, you probably need three things to be productive. You need a PC, need the software required to do your job, and you need access to various systems and the correct permissions to do what you need to do. Typical employees don't have all three of these things for weeks or sometimes months on friend of mine was not able to do her job. She didn't have the software. She needed to do her job until six months into working there. And this is very typical. So,

00:08:28

Um,

00:08:30

So why doesn't this get fixed? Well as a new employee, that's feeling the pain. You can't fix it. You're the new employee. You're, you're not empowered to fix things and experienced employees don't feel the pain anymore. They've they've got their setup, they're all working. So they don't feel the pain. So there's no incentive to fix it. Plus even if you do want to fix it, oh boy, you to fix these things, you're going have to work across so many different silos. The it department, InfoSec engineering, HR, I mean, so many different organizations are involved in the onboarding process. Could you imagine if you want to fix this problem, just having to bring all these people together to get that done. The problem is that if you don't do it, no one else will. So, uh, there's a new book, the unicorn project by gene Kim, it's the followup to the Phoenix project, which is, uh, an excellent book that kind of crystallized the dev ops movement.

00:09:29

And now, uh, gene Kim has his new book, the unicorn project, which is about this very problem. How do we make, uh, the new environment for employees, uh, workable? And, um, you might be thinking, well, yeah, Tom, that's good, but I'm going to stay at this company for a long time. So it's a little overhead, but who cares? Well, you might be at a company for many, many years, but you're probably going to be changing projects all the time. And every time you change projects, it's like being a new employee and for a company to be really dynamic, you need people to be able to change projects quickly. Plus hopefully you're always hiring and growing and you want those people to be productive as soon as possible.

00:10:10

The next thing we're going to talk about is making right easy or another way of saying it is the lazy path should guide you to the right way. So I have a couple of examples of this. Uh, the first is labor SSL versus open SSL. Now S open a cell is the technology that most websites around the world use to have, uh, a secure connection, you know, HTTPS instead of HTTP. And if you've ever used open SSL, it requires a freaking PhD to get it all configured, right? If you're going to build a secure application, there's so many knobs to turn and settings to set. And even if you do get them all right, six months later, all of those settings could be completely stale because, uh, you know, some hash algorithm is proven to be less secure than we expected, or there's a new encryption scheme or a new protocol.

00:11:05

And so open SSL, uh, is very difficult to get right or, and stay right. The LIBOR SSL fork is done by a number of people who said, let's make right easy. They came up with a new API. That is timelessly correct. So if you do the lazy thing and just say, give me, you know, listen on port 4, 4, 3, it does everything correct, and a timeless way. So, um, we can learn from this. We can embody this kind of thinking in lots of different parts of our environment. For example, at stack overflow, our CICB pipeline embodies the recommended practices. If you just go with the defaults, you're going to get all of our recommended practices. Um, another example would be the, uh, base libraries. A friend of mine at another company, uh, explained to me the problem that they were having. He said, it's difficult enough to hire a good programmer, but it's almost impossible to find a good programmer that also understands how to write code that's operational by operational.

00:12:12

I mean, you runs well on a 24 7 website, you know, collects telemetry for monitoring standard flags, uh, all these different things that are important at an operational level. Well, they embodied all of those things as the default part of their base library. So now, if you're writing a hello world, web server, you can't not have all of those good operational qualities. Um, you could diverge from their standards, but you would have to go out of your way. And you know, we're busy people. So, uh, people generally stay with the defaults. We can embody this thinking in all aspects of our foundational tools in our dev ops world, from our ticket system to our source code control system, to just how we do our operation. R O S set up.

00:13:04

Third thing I want to talk about is ubiquitous documentation. Um, and a story that relates to that is, uh, Paddington train station in London. So if you fly into Heathrow airport, there's a good chance that you're going to go from there to take a train to Paddington station. And this happened to me just last year, I arrived in Paddington station and the first thing I thought was, well, now I need to take a taxi to my hotel. And I was about to ask someone where the taxi stand is when I look down and on the ground are lines that tell me where to go for the different, uh, tube blinds and taxi stations. Let me turn the photo so you can see it a little better. See taxi go that way, the right information at the right time, I was a new arrival and there was the information I needed.

00:14:00

The information wasn't too much or too little. It didn't explain the taxi prices or the history of taxis. It wasn't the Wikipedia page on everything you could know about taxis. It was just one word and an arrow, exactly what I needed. And that's the principle I want to embody in my DevOps world. I want documentation when I need them, where I need them. And that's easy to do with a deep link URL, right? This is not like the old days where you had to specify a, uh, a book and a page number and chapter and all that kind of stuff. Um, things are so much easier now that we have the web. So let's use this technology. Let's put our URLs in error messages, in our control panels, in our alert messages, wherever people might want the information when they'll need it. You know, who gets this?

00:14:54

Apple gets this. I don't know if you've upgraded your Macintosh lately, but they've changed the default shell in the terminal. And if you're using the old shell, you get this message. It tells you we've changed the shell. Here's the command so that you could change along with us. And if you need more information, here's the URL. That explains exactly why we're doing this. So how do we create a culture of documentation? So here in my four favorite ways of creating a culture of documentation, first of all, management has to set expectations. It's not done until it's documented. And I don't mean just big projects. I mean, every little project, every time you close a ticket, you should update the documentation. So that the next person that has a similar ticket has an easier time. Second file bugs about docs, just like software, software, documentation should be treated just like software you file bugs about it.

00:15:53

Those bugs become visible. Time is allocated to fixing them, et cetera. This is very important because stale documentation is dangerous and everyone should feel empowered to update a document. And if you aren't the right person update the document, you should at least make it visible. So someone else can, that relates to creating a culture of always updating documentation. As you work, you should always be documenting. Don't think about documentation is something you do at the end of the project document, as you do your work. And lastly, you need to have good responses to excuses, to not document. For example, when a developer says, my code is the documentation, you have to say, well, you know, at least there needs to be documentation that points people to that code, give enough information so that people can get started. Another way of inspiring people who might be resistant to writing documentation is point out the fact that you'll be able to relax better while you're on vacation. Many engineers haven't taken vacation in a long, long time because they feel like they can't well, the better vague document, the more relaxed they can be while on vacation. Also, another way of thinking about it is if my stuff is really documented, well, someone else can do my job and I can go on to more interesting projects.

00:17:20

Now, some people say to me, Tom, this all sounds great, but docs, what docs my organization has like no documentation at all. And that makes sense, because, you know, if I asked everyone to raise your hand, if you like documentation, I have a feeling that not a lot of people would raise their hand and those that do well, maybe they're just raising their hand to make me feel better. So let's talk about why people avoid documentation and let's try to come up with some ways to fix that. So I think there's three big reasons. Uh, first when I know that often when I start to write a document, I get kind of flustered at the, uh, the scope. Like, what am I trying to write here? Is this the document that, um, is going to have to explain the whole history of things, or just the specific, how to do this one task.

00:18:13

And I get flustered enough that it becomes a reason to procrastinate. And then I look at my watch and it's lunchtime. I go to lunch and I never come back to this document. Similarly, I'm often uncertain what the audience is. Do I have to write this for someone who just joined the company? I have to explain every little detail, or can I just focus on the tasks that task at hand and thirdly, there's, there's a, well, there's a really scary thing. And, and I want to warn you before I turn to this next slide. This is the scariest thing to writers. This is more scary than any Stephen King, novel or movie. This is well, okay. I'll show you, but promise me that you're sitting down and ready for a scary slide. The scary thing is the blank screen. Yes. Uh, the blank screen is probably the scariest thing to writers. And, uh, just to, to drill down on this little bit, here is a blank them document for you, Microsoft windows, users. Here's what a word document looks like. Uh, for you junior people is blank notepad, a few more sophisticated people like sublime. And of course, because it's trendy, we wouldn't be, uh, we wouldn't be complete without dark mode, blank screen.

00:19:33

Um, blank screen syndrome is a real thing. It's something that writers talk about all the time and while not a recognized metal medical condition, blank screen syndrome is a very real problem. So how can we work against this? Well, first of all, we can have really awesome templates, right? In my environment, every, uh, service in our ecosystem has what we call the service dock. And it's hard to write. Well, it was hard to write until we came up with a template. And now, um, it's not like you're writing this document. You're just kind of filling out a form and you could leave certain items blank and come back to them later or let someone else fill it out. But it gets you, it gets you started and voids that blank screen. Similarly, every alert that we get in our monitoring system, we have a template. This is the stuff that gets documented about each alert. It brings consistency. And also it's very important because, you know, when you get paged at four in the morning, you're not your full self. So being able to just, uh, find something in a template and, you know, it's always, um, you know, the same place in the template for a suggested resolution. We can just start, uh, following from there.

00:20:49

Another thing you can do is encourage every one to write in small batches. I mentioned before, uh, every time you close a ticket update the documentation, so that it's easier for the next person. That's exactly what I mean by writing small batches, don't do a project and say, oh, I'll do all the documentation at the end. Keep the documentation update updating as you're going. Um, and related to that, uh, include documentation updates in work estimates. So when you tell your boss, this is going to take three days plus one day to do the documentation. You're, you're, uh, you're hurting yourself instead. Just say, this is a four day project. Don't think of documentation as this extra thing. Think of it as part of the project itself.

00:21:40

The last thing I'd like to suggest is find where engineers already, right? And repurpose that. So if someone sends you email with a great explanation of something, reply with a compliment, Hey, what a great explanation. You should put this in our document repository, then it would help everyone. They've already done the work. Now you're making it visible all over the place. I do this in email, in chat rooms, instant message. It's a great way to motivate people. And, and how many times a day do you have an opportunity to compliment a coworker? Well, here's another opportunity.

00:22:18

Um, you know, where else engineers, right? Even engineers hate to write. For some reason they love posting to stack overflow.com. And I'll tell you why it solves those problems that we talked about earlier. There's a very specific scope. Someone asks a question and you know, the scope, you have to just answer that question. You know, the audience based on the question, you kind of guess, well, is this, you, you can guess that this is a beginner or an expert, and you can target your response to that audience. And there's also, uh, templates and, uh, wizards that help you through both the question asking stage, as well as the answering stage. I mentioned a little foreshadowing, so I don't want to, you know, turn this into a big commercial, but I would like to point out that stack overflow for teams does give you that question and answer environment private for your team. And it's a great way to, uh, to, to leverage that energy that the engineers have.

00:23:25

And, um, it's also nice that new employees can fix, uh, or update questions or answers. Um, experienced employees who are maybe, uh, seeing pain. Others can take an opportunity to fix things. I met one of our customers who said he maintains this, uh, library that's used across his entire company and he never got feedback about it, but he created a tag in his personal stack overflow. Uh, that was the tag was the name of his library. And he started watching for any question tagged with that tag. And every day he just started seeing this, uh, flow of questions and it became the way that he was able to help people. And he felt so good about being in help people. Um, and it let people use this, this great library throughout the company. And we're seeing great success in it and engineering, software development, all, all sorts of areas.

00:24:30

The last thing I want to talk about is leadership. So we've talked about a lot of different things today from smart defaults to templates, et cetera, et cetera, who is going to make these things happen? Well, it's going to be you. It's gotta be you because you're the person that's concerned about these things. Now I give a lot of talks at conferences and people often come up to me afterwards and they say, yeah, you know, I liked your ideas, but I'm not a manager. I can't make this happen. Whoa, whoa, whoa, whoa, whoa, managers and leadership are two very different things. And I want everyone to be a leader. So first what's the difference management is it's a thing on the org chart. It's a thing for HR, right? Management is about setting priorities, providing resources to meet those priorities and clearing roadblocks along the way.

00:25:19

It's something that's in your org chart or your business card. But leadership leadership is something everyone can do. Leadership is about going first, clearing the way and making it easy for others to follow. I think this is so well symbolized by the child game or the kid's game followed the leader, right? That person in front, they're going first and they're making it easy for others to follow. So maybe, you know, they're, they're walking around and they get to a tree with a low hanging branch. And it doesn't matter if they're going to, uh, how they're going to get through this branch. Like they could duck underneath it, or they could lift the branch and walk under it. The decision doesn't particularly matter, but the decision has to be made. And that leader makes that decision. And that makes it easy for everyone else to follow.

00:26:09

So you can make that template and everyone else can follow along. You can, uh, build the system, maybe, you know, build half of it, but create it, uh, create the scaffolding so that other people can join along and plug in their, their addition. I also like to think of the, the stone soup metaphor. You're kind of setting up the kitchen and having everyone else bring their bits. So we've talked about a lot of things today. Um, I hope you agree that dev ops environments should strive be low context. And there's a couple of ways we can do this. We can focus on smart defaults. We can make right easy and we can make ubiquitous documentation. There are a number of impediments to writing documentation. There's the burden of the audience and the scope, you know, the blank screen syndrome. And we can fix these things with templates, templates, templates, repurpose it, repurposing texts from where engineers are already comfortable writing and providing a central source or repository repository for people to put the documentation. I hope you enjoyed my talk. Thank you for attending.