Las Vegas 2020

The Three Anti-Patterns of DevOps

There are a lot of books and papers discussing DevOps guiding principles, DevOps patterns. This is focused on some of the most frequent anti-patterns that I have seen in the past 15 years, both in consulting, but also talking to many companies in helping them understand DevOps.


The presentation will cover the three popular anti-patterns; what issues they cause and how to avoid them (or if you have implemented them, then what actions that you need to take). The goal of this presentation is to make the audience aware of common pitfalls and a guide teams that are struggling with implementing DevOps and getting value out of this change in culture (and with the associated change in processes).


Responsible for the technology implementation and support of Real-Time Payments (RTP), a 24x7 real-faster payments system for The Clearing House with staff in NY, NC and TX as well as multiple vendors in US and Europe. RTP is changing the way Payments are made in the US. Financial institutions of all sizes are taking advantage of the RTP network’s capabilities to create or enhance digital services for their corporate and retail customers.


Previously, Colin was VP and Head of the Common Services organization within the Federal Reserve Bank of New York (FRBNY). At the Federal Reserve, Common Services drove technology solutions across the Federal Reserve System (FRS) including DevOps, Cloud, NLP/AI, Engineering, Data Services including Big Data and Digital Experience.

Colin started his career at Hewlett-Packard, working in several divisions including Knowledge Systems Labs; NetMetrix and OpenView divisions. Colin has a B.S. in Computing Science from the University of Glasgow.


This session is presented by Sonatype.

CW

Colin Wynd

Head of Real Time Payments Technology, The Clearing House

Transcript

00:00:12

My name is Colin whined. And today I'm going to talk about, uh, the three popular devolves and Tapad friends. So the reason we're talking about this is, uh, over the past 10, 15 years, I've talked to lots of organizations, uh, about dev ops and, and how is our getting to dev ops. And, and there's several things that, that I've noticed that are really anti-patterns to that. And so I really want to start talking about, uh, these, and if you actually encounter them, how you actually get out of these anti-patterns and that will really help you with your migration towards that devolves, generally, people always ask me, you know, what is devolves? And, uh, you know, I think it's always good to have a definition. I've got a slightly different definition. There really helps a lot of executives understand a devil is really the phrase I have.

00:01:07

It's the ability to change a line of code and get it to production that same day with confidence. And I think the piece about it that's critical is with confidence. And that means that, you know, you gotta have the right tooling and the right processes in place to, to be able to move a line, a code, and be able to do the right levels of scans, such as security scanning, sectors, code quality, such as verifying your open source compliance. You don't have any vulnerabilities in that, even checking your open source license, to make sure that it adheres with what you're trying to do with overall application. Like you're applying also means that you have to basically tightly integrate what the developers are doing with a lot of information security needs and what operations needs. I know Mark's point and getting it to dev or QC no whole point is to get it to production.

00:02:02

And so that's my definition and that kind of frames what I'm trying to achieve and what we're trying to achieve with dev ops. And it's a slightly different definition, but hopefully that helps, you know, with overall goal with why people are actually migrating to devils. So what is an anti-pattern? So there's, there's lots of different types of Angie times. You see this and application development, uh, quite a lot. There, there's definitely things that people have done in a consistent fashion that basically Luke attractive initially, but on the long-term actually, uh, you know, do quite a bit of harm or are actually cause more work than walk. And probably the one I love talking to bio is, is people copy and paste code, right? So you might have a simple example. And the one I'll talk about is, uh, logging. So you might be able to do some custom logging and this one application, and then you start working on a second application and what do is now that was pretty good code.

00:03:01

Let me just call it and I'll start modifying the exact application. And you know, I'm working on a third application. I'll just copy it from the second application. Yeah. I made some mods, uh, but I'll copy it to this third application and I'll be really good. Not all of a sudden, right, a year later or two years later, you might have a hundred applications. And what I've found is they'll actually end up having more than a hundred copies of this because somebody else made a copy within one module of a code and it might be the same application, but you might end up having two or three options in that same application. I'm just logging and this is just logging. Uh, it could be anything. And so, you know, really the anti-pattern is there don't copy and paste, create a module trade alive or a, and basically use the library.

00:03:52

So when you update the library and the library, all of the applications either get that the new version whenever the recompile, and they can decide what they want to do as well as any new features that you could in, uh, all of the applications can benefit at a minimal cost. And that's a good example of an anti-pattern and technology, but the same thing also happens in processes. And also there's handy patterns with how people are organized. And this is what I want to talk about. And basically what you want to do is even though it might seem attractive initially to go on it or to elaborate that I have a bathroom in the long run and actually hearts. And so the idea is to try and avoid these anti-patterns and there's lots of them. I'm just going to talk today about, uh, kind of the top three anti-patterns that I've seen in the last 10, 15 years.

00:04:46

So what are these, uh, and I'm just gonna quickly go through them and then I'm going to, uh, do a bit more, a deep dive. So basically if your team is called dev ops, then you're probably not doing taboo. And this is what I see quite frequently is a lot of organizations have dev ops teams that are actually doing dev ops. But on what I see even more is actually people who have basically created dev ops teams. And most recently it tends to be the tooling team that actually implements CICB is actually known as the dev ops team. Uh, but they're not really doing dev ops. They're just a support organization, uh, for the application development teams who are actually supposed to be doing the dev ops. Number two, you need to be doing agile to be able to do a dev ops. And this is really far a lot of the companies that either have like an interesting model such as Rob are waterfall based models, who basically are getting pressured to suddenly do devil.

00:05:44

And I'm saying, don't do that jump from waterfalls straight to dev ops to the jump from waterfall into agile and then to do, to devil. And basically the third one is you're not doing dev ops if you're not going to production. And I have lots of conversations, uh, all across the U S about companies who are claiming to do dev ops, but they don't actually get to production. They do dev ops to a dev or a QC or a UAT environment bot it's a whole different process and a whole different team actually gets, uh, the software into production. So I'm going to talk about these three and a bit more detail. So if your team is called dev ops or probably not doing dev ops, and this is the typical model on the left hand side, right? You have a development organization, you have an ops organization, and basically they need to work together in very logical teams, really as a partnership.

00:06:45

And really it's this integrated team consisting of all the resources, right? Consisting of developers and operations and information security. That's really focused on creating the business value of right is really getting, uh, the business value out of the application and actually why you're doing the application development. And that's really the way that we should be done equally. What I've seen is a dev team, uh, suddenly changes his name to be devil. None of the other processes change, uh, the technology hasn't changed, the organization doesn't change. It's just a simple rename. Uh, normally of the day of team, you see also an operations team getting to be renamed to be devils. And basically that's a, a good, fantastic anti-pattern that you should really try and avoid. A lot of times it was, you know, the executive suddenly reading an article about dev ops, going to the dev managers, or are the operations manager, certainly not bull and saying, Hey, we need to do dev ops.

00:07:48

And the first thing that they did was to change the name of the team and basically misses the whole point, right? It's supposed to be an integrated, uh, unit, uh, working together very collaboratively to be able to achieve that. Second thing is you actually go and crate a separate dev ops organization. Right? I see that quite a bit, uh, where they don't actually work with that, the dev or the existing dev or the existing operations team, they actually create a whole new organization. And sometimes that might be fine if you say, okay, over time, we're going to train all of the existing developers and all existing operations. And over time, we're going to migrate them into this new dev ops organization. However, a lot of times it just caused a lot of friction. Um, and you've got a lot of passive aggressive behavior, uh, because people say, well, is this the new kid on the block?

00:08:44

They'll go to get all the funding they're gonna get all the visibility. And more frequently, what I've seen is actual tooling team, the team that actually crates, uh, the CIC D tools integrate some together in a secure fashion, they're actually known as the dev ops team. And certainly when I've been interviewing resources, you see them on the resume. A lot of people are putting, uh, you know, a dev ops engineer, uh, working on a dev ops team. I really, a lot of times they are there. They're just doing the usual, uh, you know, source code to binary repository. Y'all get, uh, Jenkins, for example, um, nexus as a typical, uh, tool stack. And they're just integrating them together, but they're not actually doing any value add, they're not actually producing, uh, any applications that go to production. And so, again, that's another one of these anti-patterns so really changing the name of a team doesn't fakes your underlying processes or technology issues, nor does it train the staff or the organization of our devils.

00:09:53

So the idea is trying to avoid doing that if you have to do it focus on, on the left, right? Um, so what you really need to do is fix your basic processes, right? You need to implement continuous integration. That really includes your automated regression tasks thing, your security scanning, uh, open source compliance and open source license monitoring, Kohler to views, et cetera, et cetera. You also need to implement continuous deployment. And I go through all the environments, not your dev or QC. How are you doing your continuous measurement? And remember the infinite loop with a dev ops, part of it that's critical is to measure, uh, your environments and your applications to give feedback back to the development and operations team or your team structured correctly. Right. Do you have all of the resources working in a single team focused on the right business outcomes, right.

00:10:53

And is there an isolate out a good balance between the development and your operations, right? What you don't want to have a development do? A takeover of operations are vice-versa where so much stuff is mandated. You're not actually getting that equal partnership. It's more of a dictatorship, right? And are you building the right culture and does management have a focus on building the right culture? Right. And this is going to be critical for us to be able to evolve and move forward to improve, uh, the speed in which we're doing, uh, uh, producing our, our code into production.

00:11:33

Second thing as a second anti-pattern is you need to be able to do an agile, to be able to do dev ops. And I talked to a lot of large organizations and a lot of them still have a lot of either waterfall or interested application development methodology. And what they suddenly decide to do is, okay, we're suddenly going to do dev ops and they tend to do a little bit of this. And a lot of times that land up just failing because they're just used to a waterfall methodology and they just can't get, uh, the processes and the people and the culture focused on doing multiple releases and you'll end up having it. Well, w we, we are doing dev ops spoke. We don't go to production. Our, we do dev ops, but we can't because, uh, you know, we ha in a highly regulated environment, so the regulators won't let us do dev ops or whatever excuse. Right. And I used to work for the fairies are bank of New York. And I worked for the clearinghouse and these are two extremely regulated organizations. And so if we can do it, um, I think just about any other company can do it, that work in a regulated industry. And really what it is is you can't do dev ops. If you work in a waterfall or an iterative model, you really gotta be doing an agile kind of process to be able to get there.

00:13:02

So how do you move to dev ops if you're in a waterfall organization and this kind of slide kind of talks about it, right? What you have to do is to start implementing agile processes. And it's very hard for organizations who are focused on waterfall to suddenly decide to do IGL, right? They've got a lot of gates and a lot of stacks, right. That they normally have to implement. You know, they tend to be very big on gathering requirements up upfront. Um, and so you tend not to have all product kind of concepts are the product is so well-defined upfront. It's very hard for them to understand. You don't need to get all of the requirements upfront and you can change the requirements I going forward. And so what, what we can to try and, uh, tell these organizations is, Hey, why don't you try and do a wa or scrum fall, right?

00:13:57

This is where you still gather your requirements up front. You still do a deployment towards the end. Uh, they're on the main and the middle, uh, with your testing organization, with your development organization, security organization, you start to do some more, uh, scrum you'll do, you know, two or three week sprints. Uh, what you'll try and do is, uh, you know, you'll do your development, then you'll do your testing, do your development, do your test bang, and then you'll eventually do a deployment. And then what your next phase would be to be more agile, like, right. So you'll still maybe gather your requirements or a good chunk, your requirements upfront. You'll still do a release Tarzan where on the middle, you'll try and get to two weeks sprints and you'll do a bit of design and a little bit of coding and then Tash thing, et cetera, et cetera, et cetera.

00:14:52

And then he'll get the next phase is even more I now, well, you'll actually try and actually get, uh, uh, a nice sprint backlog. You'll do some sprint planning and, you know, you'll actually do some design and testing, et cetera, et cetera, et cetera. And then you'll get more eyes on them, right? This may be an, uh, say, uh, a four to six month project. You'll actually try to go production, uh, two or more times, right. And this is where, okay, you're gonna start to do a sprint planning every two weeks. You'll actually start to do a coding and testing within a sprint. Um, and you'll start to do deployments, you know, maybe after two or three or four sprints, you'll start to get to there. And, and once you're in that mode, right, it will be much easier, much about our jump to go from agile to dev ops because the increment and the amount of changes in your processes and the changes that you need to do, and your organizational model is much, much smaller.

00:15:58

A lot of times I feel it's just too big of a jump, uh, to try and go from waterfall direct link to dev ops. And, and so you need to, what you do over a period of a year or two years is actually get to agile first and then go in and start to implement more of a dev ops philosophy. And that gives you time to change your culture, to change a lot of your processes that you have and organize the teams and have people be able to, uh, you know, understand dev ops and kind of get the basics in place. And I think that's going to be a critical step, uh, rather than the, the big jump. So what can you do? Right. So again, this is all about taking a step back, right? It's really rather than try and a waterfalls where you're getting into production once a year, maybe twice a year, you know, really it's a goal of trying to get production maybe every four months, you know, three times a year, then you get it down to three, three times a year at four times a year.

00:17:01

So that's every three months, et cetera, et cetera. So it's, it's these illustrative, uh, stats and slowly moving towards a dev ops, kind of big bang implementations tend not to work, right? And this gets the chance of, of your team. And everybody's mature. And allies need to look out all the blockers that you might have, whether it's information security or a governance model, and bring them into the fold to say, Hey, this actually improves our overall security posture and Rick's risk posture because we can deploy. So it's so quickly, right? So the third one is, you're not going to doubt. You're not doing dev ops, if you are not going to production. Right. And, and I see this, so it's just a fantastic anti-pattern to talk about. It's a lot of people tend to, you know, have their idyllic in place tans. A lot of times when you see this, it's the V driven primarily through the development organization where they really want to go and do dev ops.

00:18:10

Uh, but they really don't have control of the QC environment are more likely the production environment. And so you see this where development wants to try and drive dev ops. And it just, as an indication of a dev and operations team, that's not insane. And I thought it would a balance. And maybe it's one of these takeovers, uh, that I talked about earlier and really it needs to be a partnership. And you kind of see that a lot where the development team actually puts together the CEI process, so they can actually build their software, they go through and they have it that your continuous integration process integrated with our, you know, the open source compliance with, uh, all of the right pipelines doing, uh, uh, security, scanning, code quality to reviews, everything like that. Uh, the problem that we then see is they don't have enough control of either the QC environment or the production environment where it just suddenly grinds to halt.

00:19:14

And then this is where they need to work with operations to actually, they didn't get it deployed into the higher environments. They probably have control of dev. And there's enough plugins these days where you can actually deploy quite easily, uh, from your artifact or positive tray, such as nexus into a, you know, a dad, their trouble is that they don't have the rights to be able to apply this into, uh, a QCR production and operations team still isn't bought into dev ops, are they, I'm being told about it in some cases, and all of a sudden, you, you, you suddenly have to have I take it. And then they have to have lots of meetings and reviews, and all of a sudden it takes, you know, months sometimes to get into production. So if you see that it, to me, it's pretty to, obviously you're not really doing dev ops because you don't have operations bought into this.

00:20:07

And what you really need to be doing is, uh, you know, integrating, uh, all of the teams together as, as a joint collaborative app bar to avoid, uh, you know, show Stompers like this. Um, so that's, you know, one of the big things that we see, so to kinda summarize, right, you know, dev ops is really an evolution. You can't really wake up one day and say, okay, uh, everyone's going to start doing dev all set today. Right. Really need to think about, uh, it's an evolution and it's going to take time. It's not something, uh, where it's like a product on ongoing to implement it. And I have the steps laid out because I've done this 10 times before it really is an evolution, and there's going to be things that you'll do, right. And there's going to be mistakes you're going to make, and you're going to hop down.

00:21:00

And just also dev ops is different for a lot of different organizations. Your organization might be unique, right? You might not have very many constraints are, uh, rules or regulations that you have to adhere to. And so how you actually implement dev ops is going to be different from say a regulated industry where they have to prove certain things, right. You have to prove that this source goal is matches their spine, right. And it, the plight and have this nice, fantastic audit trail. And how did you achieve that also, uh, how you worry about open source licenses, uh, how you worry about, uh, vulnerability management and all of these different aspects that you really need to be careful about comparing yourself to a very different organization and say what we'll do, what Google does, uh, because you know, they seem to have it done it, right.

00:21:57

It might just not fit for your type of organization. And as E everyone's got to remember, it's not just about the technology. Uh, we get, uh, tall count by a lot of vendors saying, Hey, if you implement X product, uh, you'll be dev ops and it's not just accurate, right. Uh, the vendors are doing what they always do. Uh, but what you need to do is it's really focused on the processes that you have, and the cultural of having teams work very tightly and collaborative together with that common goal. Right? And the common goal is to get that line of code into production that same day with competence, right. And that really should be, you know, kind of a common goal. And that ties everybody together and ties information, security, ties, operation. It ties our development resources like focused on what we're trying to achieve.

00:22:52

And everyone gets to applying on that because it's with confidence and we get to define for every organization what that confidence level is. And what do we mean by confidence? Right. So a lot of times it's really focused on the processes is really looking at, you know, what manual processes do I have? Can I back away from these manual processes and uncle mind, some level of automation, uh, I'll talk about, uh, source, uh, licenses, for example, uh, there's lots of cases where you might move from one version of an source product to another, and the organization behind it might change the type of licenses, Hughes, uh, how are you going to automate the detection of that? Is that a manual process? I'll spend some time to automate that same thing, but, uh, vulnerability management. So you might have to do a code scan today and your, your source code in your libraries might be power fake.

00:23:51

No, no one, no one, uh, Barneveld lattes, uh, you get that into production. Well, it just doesn't stop there. Right. Uh, you know, vulnerability management is always just a point in time, you know, what are you doing to review the code and reuse your open source, like, uh, libraries, um, and, uh, you know, get, make sure that on a continual basis, uh, you're looking at their vulnerabilities, uh, sort of a tie back play has a great, um, you know, a tool for that. And they actually do a state of dev ops, really talks about, uh, the vulnerability management. And so I encourage everyone to, to have a look at that and have a read of that. It's a, it's really a fantastic document. Next thing is, everything is cold, right? And these days, this is the way the world is that, you know, as long as you can define everything in code, uh, you can then start to do a lot better reviews, all of that.

00:24:47

Right? So instead of having infrastructure that's custom built, and if you define your infrastructure through code, such as, uh, Terraform, for example, uh, you know, you'll be a lot better for, uh, able to move things forward and be able to make changes as needed, uh, versus worrying about, uh, custom builds, uh, even security guys call it, how are you doing your, your scanning? Uh, how are you actually implementing, uh, you know, the security controls that you have and not just your infrastructure all the way up in your application and noticed various frame marks, such as that be SIM that allow you to define, um, what security controls that, uh, for your organization would make sense. I had. The last thing is ensure you have the fundamentals in place, right? So, uh, from a process perspective, right, we, I general is the methodology, uh, that you want, whether that's scrum, uh, our pick your choice of agile methodology.

00:25:49

And, and, and it's not just scrum it's, you gotta be able to do agile and, and a very advanced way. So it's not war or scrum fall. As we see our organizations to really doing true agile, are you actually following all of, all of the rules and all of the guidelines, the agile hats, next one is CI CD, right. Do you actually have that as an automated process, right. Can your developers decide to do a build to any time? And does that build actually integrate into say a pipeline system, such as Jenkins, for example, and does it integrate and kick off other tools such as, uh, SAS, uh, our DAS security scanning code quality, and then where, uh, uh, you know, the resulting binary's put in where's your artifact repository, and then how do you do the deployments are of that? And really that's key.

00:26:47

And, and a lot of that is underlying products. There's open-source products, and then there's purchase products on top of that. And I think that's one of the key aspects of the fund. The fundamentals of dev ops is agile and CD, a CIC D, and then the last piece is monitoring, right? And a lot of people forget about the monitoring, uh, but really it really needs to inform not only the operations, uh, Sapper, also the development, right? Where are they shoes? Can we tell when there is an issue, you know, when the several is done, that's pretty, uh, uh, standard these days, but more advanced you can, can you actually determine when your API goes down, are you doing a health check on each of your API calls to see what might be the issue? And they send you correlate them together to say, yes, let's do an automatically star if, uh, the API, uh, is failing.

00:27:42

Um, and then if it doesn't restart, you know, automatically submit a trouble ticket and do some form of escalation and all through an automated fashion. And so these are kind of, to me, the baselines that people need to have, uh, put in place before trying to advance any farther. Cause if you don't have them, what are you going to have? Is you already start doing some site hacks, whether it's a process hack to bypass it, um, or some, some people hack by saying, well, we'll just be it on operations, or we'll just be it on the development team. And this just fails, right? It's over a period of time. As you start to throw the whole dev ops culture, it starts to break down. And this is the whole point of me talking about the anti-patterns is to try and avoid all of that. So hopefully this has been helpful. Uh, again, my name's Colin wine and I've been talking about dev ops and Islay implementing dev ops for probably the last decade. So thanks and, uh, feel free to reach out to me if you have any questions. Okay, bye.