The Three Anti-Patterns of DevOps

My name is Colin Wynd, and today I'm going to talk about the three popular DevOps anti-patterns. The reason we're talking about this is, over the past 10, 15 years, I've talked to lots of organizations about DevOps and how they're getting to DevOps. There are several things that I've noticed that are really anti-patterns to that. I really want to start talking about these and, if you actually encounter them, how you get out of these anti-patterns. That will really help you with your migration towards DevOps.

People always ask me, "What is DevOps?" I think it's always good to have a definition. I've got a slightly different definition that really helps a lot of executives understand DevOps. The phrase I have is: the ability to change a line of code and get it to production that same day with confidence. The piece about it that's critical is "with confidence." That means you've got to have the right tooling and the right processes in place to be able to move a line of code and do the right levels of scans, such as security scanning, code quality, verifying your open source compliance, checking that you don't have vulnerabilities, and even checking your open source license to make sure that it adheres with what you're trying to do with the overall application that you're deploying.

It also means that you have to tightly integrate what the developers are doing with what information security needs and what operations needs. There's not much point in getting it to dev or QC. The whole point is to get it to production. That's my definition, and that frames what I'm trying to achieve and what we're trying to achieve with DevOps. It's a slightly different definition, but hopefully that helps with the overall goal of why people are actually migrating to DevOps.

So what is an anti-pattern? There are lots of different types of anti-patterns. You see this in application development quite a lot. There are definitely things that people have done in a consistent fashion that look attractive initially, but in the long term actually do quite a bit of harm or cause more work than wanted.

Probably the one I love talking about is people copying and pasting code. You might have a simple example, and the one I'll talk about is logging. You might be able to do some custom logging in one application, and then you start working on a second application and say, "Well, that was pretty good code. Let me just copy it and I'll start modifying it in the second application." Then, working on a third application: "Yeah, I'll just copy it from the second application. Yeah, I made some mods, but I'll copy it into this third application and that'll be really good." All of a sudden, a year later or two years later, you might have 100 applications. What I've found is you'll actually end up having more than 100 copies of this, because somebody will have made a copy within one module of code, and it might be the same application, but you might end up having two or three versions in that same application of just logging. And this is just logging. It could be anything.

So really, the anti-pattern is there. Don't copy and paste. Create a module, create a library, and use the library so when you update the library, all of the applications either get the new version whenever they recompile and they can decide what they want to do, as well as any new features that you put in. All of the applications can benefit at minimal cost. That's a good example of an anti-pattern in technology. The same thing also happens in processes, and also there are anti-patterns with how people are organized. This is what I want to talk about. Even though it might seem attractive initially to go and adhere to or leverage that anti-pattern, in the long run it actually hurts you. The idea is to try and avoid these anti-patterns. There are lots of them. I'm just going to talk today about the top three anti-patterns that I've seen in the last 10 or 15 years.

What are these? I'm going to quickly go through them, and then I'm going to do a bit more of a deep dive. Basically, if your team is called DevOps, then you're probably not doing DevOps. This is what I see quite frequently: a lot of organizations have DevOps teams that are actually doing DevOps. But what I see even more is people who have basically created DevOps teams. Most recently, it tends to be the tooling team that implements CI/CD is known as the DevOps team. But they're not really doing DevOps. They're just a support organization for the application development teams who are actually supposed to be doing the DevOps.

Number two: you need to be doing Agile to be able to do DevOps. This is really for a lot of the companies that either have an iterative model such as RUP or waterfall-based models, who are getting pressure to suddenly do DevOps. I'm saying: don't do that jump from waterfall straight to DevOps. Do the jump from waterfall into Agile and then to DevOps.

The third one is: you're not doing DevOps if you're not going to production. I have lots of conversations all across the U.S. about companies who are claiming to do DevOps, but they don't actually get to production. They do DevOps to a dev, QC, or UAT environment, but it's a whole different process and a whole different team that actually gets the software into production. I'm going to talk about these three in a bit more detail.

If your team is called DevOps, you're probably not doing DevOps. This is the typical model on the left-hand side. You have a development organization, you have an ops organization, and basically they need to work together in very logical teams, really as a partnership. It's this integrated team consisting of all the resources, consisting of developers and operations and information security, that's really focused on creating the business value. It's really getting the business value out of the application and actually why you're doing application development. That's really the way that we should be doing it.

Equally, what I've seen is a dev team suddenly changes its name to be DevOps. None of the other processes change. The technology doesn't change. The organization doesn't change. It's just a simple rename, normally of the dev team. You also see the operations team getting renamed to be DevOps. That's a fantastic anti-pattern that you should really try and avoid. A lot of times it was the executive suddenly reading an article about DevOps, going to the dev manager or the operations manager, certainly not both, and saying, "Hey, we need to do DevOps." The first thing that they did was change the name of the team. It misses the whole point. It's supposed to be an integrated unit working together very collaboratively to be able to achieve this.

The second thing is you actually go and create a separate DevOps organization. I see that quite a bit, where they don't actually work with the existing dev or the existing operations team. They create a whole new organization. Sometimes that might be fine if you say, "Okay, over time we're going to train all of the existing developers and all the existing operations, and over time we're going to migrate them into this new DevOps organization." However, a lot of times it just causes a lot of friction, and you get a lot of passive-aggressive behavior because people say, "Well, this is the new kid on the block. They're going to get all the funding. They're going to get all the visibility."

More frequently, what I've seen is actual tooling teams, the team that creates the CI/CD tools and integrates them together in a secure fashion, are known as the DevOps team. Certainly, when I've been interviewing resources, you see them on the resume: a lot of people are putting "DevOps engineer working on a DevOps team." Really, a lot of times they're just doing the usual source code to binary repository, probably Git, Jenkins, for example, Nexus as the typical tool stack, and they're just integrating them together. But they're not actually doing any value add. They're not actually producing any applications that go to production. Again, that's another one of these anti-patterns. Changing the name of a team doesn't fix your underlying processes or technology issues, nor does it train the staff or the organization about DevOps. The idea is to try and avoid doing that.

If you have to do it, focus on the left. What you really need to do is fix your basic processes. You need to implement continuous integration. That really includes your automated regression testing, your security scanning, open source compliance, open source license monitoring, code reviews, et cetera. You also need to implement continuous deployment, and that goes to all the environments, not just dev or QC. How are you doing your continuous measurement? Remember the infinite loop with DevOps. The part of it that's critical is to measure your environments and your applications to give feedback back to the development and operations team.

Are your teams structured correctly? Do you have all of the resources working in a single team focused on the right business outcomes? Is there actually a good balance between development and operations? What you don't want to have is development doing a takeover of operations or vice versa, where so much stuff is mandated that you're not actually getting that equal partnership. It's more of a dictatorship. Are you building the right culture? Does management have a focus on building the right culture? This is going to be critical for us to be able to evolve and move forward to improve the speed in which we're producing our code into production.

The second anti-pattern is you need to be doing Agile to be able to do DevOps. I talk to a lot of large organizations, and a lot of them still have a lot of either waterfall or iterative application development methodologies. What they suddenly decide to do is, okay, we're suddenly going to do DevOps. They tend to do a little bit of this, and a lot of times it'll end up just failing because they're used to a waterfall methodology and they just can't get the processes and the people and the culture focused on doing multiple releases.

You'll end up with, "Well, we're doing DevOps, but we don't go to production," or, "We do DevOps, but we can't because we are in a highly regulated environment, so the regulators won't let us do DevOps," or whatever excuse. I used to work for the Federal Reserve Bank of New York, and I worked for The Clearing House, and these are two extremely regulated organizations. If we can do it, I think just about any other company can do it that works in a regulated industry. Really what it is, is you can't do DevOps if you work in a waterfall or an iterative model. You really have to be doing an Agile kind of process to be able to get there.

How do you move to DevOps if you're in a waterfall organization? This slide talks about it. What you have to do is start implementing Agile processes. It's very hard for organizations who are focused on waterfall to suddenly decide to do Agile. They've got a lot of gates and a lot of steps that they normally have to implement. They tend to be very big on gathering requirements up front. You tend not to have a product kind of concept, or the product is so well-defined up front that it's very hard for them to understand you don't need to get all of the requirements up front, and you can change the requirements going forward.

What we tend to tell these organizations is, "Hey, why don't you try and do Water-Scrum-Fall?" This is where you still gather your requirements up front, you still do a deployment toward the end, but in the middle, with your testing organization, development organization, and security organization, you start to do some more Scrum. You'll do two- or three-week sprints. What you'll try and do is you'll do your development, then you'll do your testing, do your development, do your testing, and then you'll eventually do a deployment.

Your next phase would be to be more Agile-like. You'll still maybe gather your requirements, or a good chunk of your requirements, up front, and you'll still do a release toward the end, but in the middle you'll try and get to two-week sprints, and you'll do a bit of design and a little bit of coding and then testing. Then the next phase is even more Agile. You'll actually try and get a nice sprint backlog. You'll do some sprint planning, and you'll actually do some design and testing, et cetera.

Then you'll get more Agile. This may be in, say, a four- to six-month project. You'll actually try to go to production two or more times. This is where you're going to start to do sprint planning every two weeks. You'll start to do coding and testing within a sprint, and you'll start to do deployments maybe after two or three or four sprints. You'll start to get there. Once you're in that mode, it'll be a much easier, much better jump to go from Agile to DevOps because the increment and the amount of changes in your processes and the changes that you need to do in your organizational model is much, much smaller.

A lot of times I feel it's just too big of a jump to try and go from waterfall directly to DevOps. What you do over a period of a year or two years is actually get to Agile first and then start to implement more of a DevOps philosophy. That gives you time to change your culture, to change a lot of your processes that you have, and organize the teams and have people be able to understand DevOps and get the basics in place. I think that's going to be a critical step rather than the big jump.

What can you do? Again, this is all about taking a step back. Rather than trying, in waterfall, where you're getting to production once a year, maybe twice a year, really it's a goal of trying to get to production maybe every four months, three times a year, then get it down to three times a year, four times a year, every three months, et cetera. It's these iterative steps and slowly moving towards DevOps. Big bang implementations tend not to work. This gives your team and everybody the chance to mature, and allows you to look at all the blockers that you might have, whether it's information security or a governance model, and bring them into the fold to say, "Hey, this actually improves our overall security posture and risk posture because we can deploy so quickly."

The third one is you're not doing DevOps if you're not going to production. I see this so frequently. It's a fantastic anti-pattern to talk about. A lot of people tend to have the right tooling in place, and a lot of times when you see this, it's driven primarily through the development organization where they really want to go and do DevOps, but they really don't have control of the QC environment or more likely the production environment. You see this where development wants to try and drive DevOps, and it is an indication of a dev and operations team that's not in sync and is out of balance. Maybe it's one of these takeovers that I talked about earlier. Really it needs to be a partnership.

You tend to see that a lot where the development team puts together the CI process so they can build their software. They have your continuous integration process integrated with open source compliance, with all of the right pipelines doing security scanning, code quality reviews, everything like that. The problem that we then see is they don't have enough control of either the QC environment or the production environment, where it just suddenly grinds to a halt. This is where they need to work with operations to actually get it deployed into the higher environments.

They probably have control of dev, and there are enough plugins these days where you can deploy quite easily from your artifact repository, such as Nexus, into dev. The trouble is that they don't have the rights to be able to deploy this into a QC or production, and the operations team still isn't bought into DevOps, or they haven't been told about it in some cases. All of a sudden, you have to have a ticket, and then they have to have lots of meetings and reviews, and all of a sudden it takes months sometimes to get into production. If you see that, to me, it's pretty obvious you're not really doing DevOps because you don't have operations bought into this. What you really need to be doing is integrating all of the teams together as a joint collaborative effort to avoid showstoppers like this. That's one of the big things that we see.

To summarize, DevOps is really an evolution. You can't really wake up one day and say, "Okay, everyone's going to start doing DevOps today." You really need to think about it as an evolution, and it's going to take time. It's not like a COTS product, where I'm going to implement it and I have the steps laid out because I've done this 10 times before. It really is an evolution, and there are going to be things that you'll do right, and there are going to be mistakes you're going to make, and you're going to have to adjust them.

Also, DevOps is different for a lot of different organizations. Your organization might be unique. You might not have very many constraints or rules or regulations that you have to adhere to. How you actually implement DevOps is going to be different from, say, a regulated industry where they have to prove certain things. They have to prove that this source code matches this binary, and it was deployed, and have this fantastic audit trail. How do you achieve that? Also, how do you worry about open source licenses, vulnerability management, and all of these different aspects? You really need to be careful about comparing yourself to a very different organization and saying, "We'll do what Google does because they seem to have it done all right." It might just not fit for your type of organization.

Everyone has to remember: it's not just about the technology. We get talked at by a lot of vendors saying, "Hey, if you implement X product, you'll be DevOps." It's not accurate. The vendors are doing what they always do. What you need to do is really focus on the processes that you have and the culture of having teams work very tightly and collaboratively together with that common goal. The common goal is to get that line of code into production that same day with confidence. That really should be a common goal, and that ties everybody together. It ties information security, it ties operations, it ties our development resources, really focused on what we're trying to achieve. Everyone gets to opine on that because it's "with confidence," and we get to define for every organization what that confidence level is and what we mean by confidence.

A lot of times it's really focused on the processes. It's really looking at: what manual processes do I have? Can I back away from these manual processes and implement some level of automation? I'll talk about open source licenses, for example. There are lots of cases where you might move from one version of an open source product to another, and the organization behind it might change the type of licenses used. How are you going to automate the detection of that? Is that a manual process? Spend some time to automate that.

Same thing with vulnerability management. You might do a code scan today and your source code and your libraries might be perfect: no known vulnerabilities. You get that into production. Well, it just doesn't stop there. Vulnerability management is always just a point in time. What are you doing to review the code and review your open source libraries, and make sure that on a continual basis you're looking at your vulnerabilities? Sonatype actually has a great tool for that, and they actually do a State of DevOps. It really talks about vulnerability management, and I encourage everyone to have a look at that and have a read of that. It's a fantastic document.

Next thing is everything as code. These days, this is the way the world is. As long as you can define everything in code, you can then start to do a lot better reviews of that. Instead of having infrastructure that's custom-built, if you define your infrastructure through code, such as Terraform, for example, you'll be a lot better able to move things forward and make changes as needed, versus worrying about custom builds. Even security as code: how are you doing your scanning? How are you actually implementing the security controls that you have, and not just your infrastructure, all the way up in your application? There are various frameworks such as BSIMM that allow you to define what security controls for your organization would make sense.

The last thing is ensure you have the fundamentals in place. From a process perspective, Agile is the methodology that you want, whether that's Scrum or your choice of Agile methodologies. It's not just Scrum. You have to be able to do Agile in a very advanced way. It's not Water-Scrum-Fall, as we see organizations do. Are you really doing true Agile? Are you actually following all of the rules and all of the guidelines that Agile has?

Next one is CI/CD. Do you actually have that as an automated process? Can your developers decide to do a build at any time? Does that build actually integrate into a pipeline system such as Jenkins, for example? Does it integrate and kick off other tools such as SAST or DAST, security scanning, code quality? Where are the resulting binaries put in? Where's your artifact repository? How do you do the deployments out of that? That's key. A lot of that is underlying products. There are open source products, and then there are purchased products on top of that. I think one of the key aspects of the fundamentals of DevOps is Agile and CI/CD.

The last piece is monitoring. A lot of people forget about the monitoring, but it needs to inform not only the operations staff, but also the development. Where are the issues? Can we tell when there is an issue? When the server goes down, that's pretty standard these days, but more advanced here: can you actually determine when your API goes down? Are you doing a health check on each of your API calls to see what might be the issue? Can you correlate them together to say, "Yes, let's do an automatic restart if the API is failing," and then if it doesn't restart, automatically submit a trouble ticket and do some form of escalation, all through an automated fashion?

These are, to me, the baselines that people need to have put in place before trying to advance any further. Because if you don't have them, what you're going to have is you'll start doing some side hacks, whether it's a process hack to bypass this, or some people hack by saying, "Well, we'll just beat on operations," or, "We'll just beat on the development team." This just fails. Over a period of time, as you start to grow the whole DevOps culture, it starts to break down. This is the whole point of me talking about the anti-patterns, to try and avoid all of that. Hopefully this has been helpful. Again, my name is Colin Wynd, and I've been talking about DevOps and actually implementing DevOps for probably the last decade. Thanks, and feel free to reach out to me if you have any questions. Okay, bye.