Las Vegas 2020
Challenges to Implementing Database DevOps
Challenges to Implementing Database DevOpsm -- Getting going in DevOps can be difficult at best of times. Throw in the need to manage data on top of that and the challenges grow. Using a case study, this session will provide you with tips and hints at how best to deal with your own database DevOps implementation. Whether your team is deploying locally to on-premises databases, or implementing cloud-based databases and DevOps technologies, the knowledge you gain from this session will help you deal with data in your DevOps. The challenges that data and databases present to DevOps are real, but they are also surmountable. Find out how by joining this session.
This session is presented by Redgate.
SA
Stuart Ainsworth
Service Reliability Engineer, Jack Henry
GF
Grant Fritchey
Product Advocate, Redgate Software
Transcript
Hello, and welcome to challenges to implementing database DevOps. My name is grant Frichy. I worked for red gate software, and this is my contact information. If you want to get in touch with me after today, I'd love to help you out. If I could, with me today is Stewart Ainsworth. Stuart, can you introduce yourself please?
I I'm still at Ainsworth. I'm a senior manager of service, reliability engineering for gladiator technology, a profit star solution, a Jack Henry company. So, uh, just to give a little bit of background, I am a, I'm a former database guy, but I've started doing a lot more with networking and infrastructure and application support, uh, in this role. And so I'm excited to talk to you guys about dev ops today.
Oh, oops. There we go. All right. So we're all up and running. So Stuart, why don't you tell me about your dev ops implementation? Let's let's start with that.
I'm sure. So gladiator technology, the, the business unit that I'm work for, we've been around since 2002. So we've been around for about 20 years now, uh, getting close to 20 years and we've got a lot of legacy applications and products. We do, uh, firewall monitoring, network IPS monitoring for funding showing institutions. Um, so we're in the security space, the managed security provider space, and we get a lot of data flowing through this application that we've grown organically over the last 18 to 20 years or so. Um, what we started noticing probably over the last five years is that, uh, you know, as this application has grown, it's gotten a lot harder in and more difficult to get new features out the door, uh, while still managing some of the technical debt that we had to, to build up inside the applications. So we really began to focus on, uh, figuring out the deployment pipeline.
A few years ago, we started looking at tools, et cetera, that can help us with that. And so we began to, we began the dev ops journey. Um, and I think that a lot of people know that once you start that journey and you never really finished it, right? I mean, it's, it's, it's just moving in place. Right. Um, and so where we're at now is really, uh, you know, I tell people that when you're in, in the dev ops movement, you're always in a point of pain, right? That, that it's never going to get. It gets easier in some ways, but the, the beauty of this whole philosophy, the beauty of this whole movement is that you start discovering new challenges as you go along. And so it's not for the faint of heart. And so where we're at right now is, is we're probably about 50% of our data footprint is being managed in a continuous integration fashion.
And there's still some improvements that we're going to make there. And probably the other 50% is not. And when I'm talking to 50% a week, we manage about 23 to 24 terabytes of data. Uh, and it, and it rotates out pretty frequently. Uh, that's a 90 day window of data that we're, that we're managing with Aaron. So, um, it's, it's, it's a lot of, a lot of logs. Ones are a lot of turnover, a lot of trainings yet. Um, there are several new features that we continue to roll out frequently. Uh, we've really tried to accelerate that particularly over the last 18 months or so with improvements for our customer basis, et cetera. And, uh, it's been interesting. So again, part of the problem is, is that you want to talk about what you're doing in an exciting and interesting way, and there's so much going on that it's, it's hard at times to say these are the one thing that I could, that I can pick out and tell you about. But,
Well, I mean, so, so I mean the, the topic is challenges, right? So it's, what's hard about doing databases. I mean, what I mean, dev ops is, you know what I mean, we talk about dev ops and it can get kind of weird and people, people get kind of odd about the whole term and, and what it means. Um, but fundamentally, you know, it it's process in support of people using tools and automation, I think is a decent summary, right? So what, what, why is why, why our database is lagging behind? Why is it so hard to, to implement this in databases?
You know, I think the biggest problem is obviously to the issue state, right, is that, you know, when you start talking about applications, uh, applications being, uh, state less in a sense that, that they can just, um, they are what they are, right. And so you can generate the code. It's going to take the moment of now and run with an and database is, you know, you have history and, uh, and if you make any change because your schema is so intertwined with the actual data that it represents, uh, you know, if that change, there's a risk associated with making that change, then if it breaks something, then you've got to figure out how do you migrate from where you were previously to where you're going to go forward with? Uh, it also is, I think it's a, I think it's a tool and cultural thing as well, that, um, you know, as an industry, we database management pulled away from a lot of the other, uh, development and administration efforts because we became so focused on just the concept of data that we consolidated both of those efforts together. And what we've seen is particularly in the development space, the, the, the tools in, and, um, uh, technologies have really accelerated and we've pretty much stayed focused on, you know, writing better SQL, uh, you know, doing query, tuning it performance management and understanding the relationships there. Uh, and so, you know,
To be fair, H a D R um, you know, H you know, the whole, the whole idea of, of what you ensuring that we've got, you know, uh, um, you know, disaster recovery becomes like, you know, pretty, uh, any good paranoid, Def professional has got a good Dr. System set up, and they've probably got a good high availability system set up. Um, and so we've spent a lot of time in there, but now we're finding that we're lagging behind, you know, in the whole, you know, um, um, you know, infrastructure as code and all the rest of the stuff.
Right. And it's, you know, and I think it's, it's far as challenges when it comes to, to, to actually spinning this sort of stuff up is, is that, uh, it's, it's that mindset of I'm here to protect the data and protect the industry, the data, right. And so we tend not to look outside of, of, of what the application is actually doing with the data. Um, I think at times that particularly in the database space, we get so focused on making sure that we, uh, are optimizing that we don't even stop to ask, do we really need this right. Replication really easily? And so it's that, it's that stepping back in, in having that holistic view over the entire application to, uh, to server pipeline? Sure. That I think that database folks like us have struggled to grasp, because we're so focused on making sure that what we're doing is being done right. And it's important, those skills I don't want to, I don't want to say that those skills aren't important and that, that focus isn't
What, no, I agree.
You gotta be able to put it in the context of the bigger picture, that the reason why you are protecting the data and the reason why you're, you're, you're protecting the industry is because of that whole value stream. And so, um, so I think that there's, there's challenges both on the tool side, right? That in there's challenges in dealing with state, and then there's challenges in dealing with culture and mindset and people and addressing all of those, um, is a struggle, right? I mean, sure. And you, and I don't, I don't think anybody can take them on all at once. I think that's the other piece of it is, is that, you know, when we talk about dev ops, there's so much fantastic literature out there right now. And so many great things going on that we, that the temptation is, is, Hey, I'm going to sit down and I'm going to do this, and I'm going to do all of it.
And then when you fail, you're like, man, this just steak, I'm going to be done with it. I'm not going to do it. I'm going to, I'm just going to throw it all away and start up and stop what I'm doing. Reality failure is part of the learning experience. Absolutely. Um, you have to try, you know, I, I always talk about success is incremental start small, and just keep focusing on one thing at a time and move the ball forward a little bit at a time. And if that works great, and if it doesn't, well, then you can go back and tweak. Just that one thing don't, don't focus on trying to do the whole big picture at once.
Well, that, that brings up my, my question would be, you know, I mean, cause, cause obviously you're not going to start with, well, we've got all of our SQL server and Oracle instances inside of containers and we've got Kubernetes and now we're able to simultaneously, you know, spin these things up and spin them down and, and, and deploy from, from an automated process. Nobody starts there. Right. Right. You know, whether or not you in there. Cool. But you don't start there. Um, where do you start? I mean, because cause, cause as you say, I mean, you know, the challenges are real, right? That the, the technologies is different, um, persistence and state matter. Right. You can't simply throw the database away. So, so where, where did you guys start in implementing your process?
Uh, the first step and this is, and again, we, so we, we picked a couple of pilot databases that we were interested in that these are the things that we were like, these are the projects that we're going to be accelerating and moving forward with. Um, and it was a mixture of both legacy and new, uh, kind of Greenfield development. Like we were spinning up a new application. So we had some new database databases that would be built from star. And so, so we've got a little bit of exposure in both of those areas and about the challenges about them. The first thing that we did really was focused on source control and you know, I think that that is so fundamentally important. And that's one of those challenges that I think that, um, I've seen a lot broader embrace of that in the last, probably five years within the database community. But I still think that it's, it's, it's, it's still one of those things that you, you want to get on the checklist, right? Is, is, are you guys doing source control? Are you source controlling your schema? Are you source controlling, um, your lookup tables, the, those, those, uh, static values that you have to have to progress forward? Is that all part of it? Right. So that to us was really the first fundamental step that we had to do. Um, uh, and you know,
Were there any unique challenges in, in just that first step?
Yes.
You know, again, I think that's, uh, uh, the successes we've had really is that, uh, you know, we had to the people that are the most familiar with source control tools and the source control mentality and the source control model are not the people who have the strengths in, in performance tuning and writing good PC full code. And so getting those two perspectives together at a table so that you could build a project inside of visual studio, just like any other sort of project that you're going to build an application in and then having it run well, once it landed on a database that was probably one of the first big heartaches. And then you have to talk about things like, um, you know, I'll give you an example, index maintenance, right? So wherever you're in. So our is indexes. You consider that part of your operation or is that part of your development path? And it's one of them, I have
An opinion on it,
You know, and you know, and I think what you have to do ultimately for us, it was, it was a lot of juggling back and forth to figure out, right, because we were, you know, uh, one of the problems when you're dealing, particularly with databases that scale up significantly more than anything that your development environment can handle. Like, you know, you, you, you have, you know, maybe a hundred thousand rows in your development environment when you start talking about millions and billions of rows, uh, in a production environment is, is that your indexing strategy is going to be effected by that. And so, you know, it, it is very tough to test those kinds of performance differences. And so what we started to do is to really have to figure out a way to handle getting those index ideas and suggestions that we've optimized for and designed for production back into the development pipeline, right?
And for a long time, it was, you know, our DBA would do something and then he would tell the developers and they would include a part of the script. So they wouldn't get fallen away and getting interviewed, getting rewritten over and over. And that's an okay model. What we've gotten really good at lately is, uh, we don't even, we will, we may make the change in production to test and then we'd roll it back out immediately and then hand it over to dev and, and make it part of an official release pipeline because you know, it's much easier to do it that way for us, because it's one way to ensure that any changes that we make in production get communicated back to the dev source control model. Right. And lock them to source control. Because I think that first mindset and mental hurdle that is have to get past is, is that the database is not true, right. Control for it. Right.
That's the big one. Right. I mean, if it's not in source control, it doesn't exist is really hard for DBA because it's like, no, no, no, no, it's in the database right there. I could see it.
Right. So, uh, you know, so getting past that and getting people to understand that, okay, yes, we want to make change, then we'll do it. And we had some, uh, we had some setbacks early on because we would do things like we would build indexes and a DBA would forget to notify Bev. And so dad would push out a release and the index would be gone. And so, and then all of a sudden we're troubleshooting the same thing over and over and over again. And, uh, really the only way to do that is to just keep pounding on it and banging on it and check it and finding ways to, uh, check for differences in who made the change. When, so, uh, we started, we started looking at doing various auditing tools to see when changes were and that worked okay. Particularly for the smaller databases.
When you start getting into the larger database has been auditing itself kind of becomes a heavy, you know, because we have such a change rate that's associated with it. So it really is just that pattern of, of, of getting the process over and over again, that you have, whatever changes you make has got to be communicated and stuffed back in towards control or else it will go away and that beyond. And, and that's, uh, I think that's a pretty common experience between people that are on the operation side, on the development side. The other struggle that we face is that because our database so large and so organically grown over the last 18 years. So there was a lot of interplay between applications and application views, multiple databases, databases, themselves would have crossed on queries and would talk to each other. Uh, and what we would find is that some of the sample databases that we had chosen to do a pilot program that I'm saying, oh, we're going to keep this in source control.
Uh, we started using SQL change automation, uh, to deploy those. We also have some databases that were still being deployed by scripts. And what we noticed was is that there were conflicts there coming out of development. We had a couple where, you know, they were not having that final merge conversation about I've had a developer change, the store procedure, and oh, by the way, it's gone on and evolve and it's stored in source control. So we had a couple of instances where things were getting overwritten back and forth, where we deploy change one way. And then we deployed another way. And ultimately, because I'm on the operations side of the house, I finally had to, uh, just come back to them and say, look, the only way we're going to make this work is if we rally around one approach, right. And you know, I, I don't really care, uh, which approach you use.
I just need it consistent. Um, and you know, and I think they, I think they got it. I think there's still some resistance because, you know, at times the tooling it's still uncomfortable for right. For, for, for folks. And so, but I started talking about, you know, that, that look, if we can get source control done, and we can have those murders conversations done and we can trust the fact that the databases and source control. And then if everything that we do is coming out of source control, uh, and everything is coming to us in a consistent fashion, then the next step in the pipeline is we can start now looking at well, how do we begin to make this much more push button, much more automated and begin deploy it, not only in the production environment, but also looking at what we can do with continues in continuous integration into, uh, a QA environment or into another test environment. Right.
So you'd say that was like, so, so we spent a little bit of time on it, although I think it's important source control, right? Step one, step two. See I, or continuous integration
For us. Yes. And, and probably, I guess it would say step one a would be using to build process. Right. You know, because I think that that, uh, uh, what we, we've moved into Azure dev ops as our, um, um, uh, source control workflow management system. And that tool is like, many of the Microsoft tools is getting so big and so broad, and there's so much cool stuff in it that I don't even really know what to call it anymore. It just does what you need to do. But, um, but we had to get people accustomed to now starting to do builds on a daily basis. Right? So you do source control, you start building things on a daily basis. At the end of the day, you check in your code and it automatically builds for you. It passes or fails just like you do an application.
Now, if you do that, you can either get the output in forms of a file that you can kind of pass the file around and say, you know, here's a, here's a package, right. And I'm not even sure of what people call those, but I like to get back. I do get a lot of people call a new get yeah. Now, but you'll get an idiot. Right. And I can, and I can take that new bit and I can do things with it. And operations. The other benefit is, is that you can actually begin to build a feed. Uh, and then other two tools can actually consume the feed. So even get one more step out of the process, I don't have to pass files around. Now I have to do this point a, a CI tool, like octopus at it. And it just says, oh, there's a new item in the feed.
Would you like to build a release? And yes, we would. And then you can begin to push it. Right. Um, and there's so much power in, in that, because all of these things that I used to spend, you know, a day, you know, okay, I've got to go, I've got to go prep the file to have, to build a test database in productions of their component compared to schemas. For sure. I have to, you know, begin to, uh, you know, copy, deploy, run all these scripts. Sometimes it's, uh, sometimes it's 30 scripts. Sometimes it's 10 scripts. Sometimes it's, it's, you know, one or two, and it's all been convinced now. And you know, that was probably a day's worth of prep work just to get ready for deployment. And then you actually push the button and pray.
Now, now I've got all of that prep work down to where it's like, oh, you've done a daily build. I can go get that version and I can push a button and it'll go from the build and it automatically begins to deploy it. It'll do all the comparisons for me. I can see if it's something that I want to approve. If I want to approve it. Boom. I can go in a way we go. Right. Right. There's so much power in that. And again, it's, it's it starting incrementally start with source control, get used to the bills, start looking in a CICB tool, try to figure out those challenges to get from one or the other, but the important stuff that you should not overlook when you're doing all of this is also the importance of making sure that your backup and recovery strategy is just as fast as your deployment strategy.
Right. And so one thing that we've had to really think about, uh, how we're going to do this, particularly, uh, when you start talking about those first early steps, like once you've done that first deployment using a nougat or some other, or some other strategy where you're pulling directly from source control, you know, what do you do if that goes wrong? I mean, so you've to, you've got, gotta have some of that old fashioned method of, of, okay, I've got a side-by-side database installation. I can see where the changes in the scheme of where let's quickly build something and push it forward that way. But you need to stop and think through all of those strategies as well. And that's, I think that's other big challenges is just getting started.
Yeah. I mean, yeah. I mean, it's easy to say step one source control, but, but there's just so much there. Um, I'm just checking the time. Oh, we got plenty of time. Um, let me ask you another question total sidetrack. I mean, because, cause we've talked technology for a bit, um, from a personnel standpoint, uh, Redgate did a survey and we do a survey every year on the state of database dev ops. Um, and the one thing that shocked me was, you know, we always talk about culture and, and the challenge is culture. The challenge is culture. And the number one challenge is reported to us by the survey was not culture. That was like number three, the number, yeah. The number one challenge was technology, but are you ready because of training because of turnover and people who don't know how to script PowerShell or don't know octopus or don't know AWS build or, you know, whatever tool that we're, we're talking, it, it's not so much the tools themselves. Um, but, but the, the training aspect, the knowledge aspect of it, how did you guys deal with that?
I'll admit that it's still a struggle for us a lot and it's for us, it's not, I think when you think turnover, you know, I immediately think in terms of turnover of human resources, but you know, it's, it's the turnover of the technology itself, right? Everything is accelerating so fast that it, it's tough to kind of get a grasp around it, um,
All of the above, um, because it's not even if people are quitting your company in droves and hopefully they're not, but it's, you know, but you just move from one project to another and the second project is doing something different, right. Even subtle differences. And, and, you know, it's like, oh, well, you know, we're doing exactly the same thing as you do, but we're not using octopus. We're using Azure. Right. And suddenly it's like, I've got to learn new stuff,
You know, for us it, and we're still pretty heavily siloed in terms of, uh, we have a development department, we have a QA department, we have, um, you know, my team, the SRAs, uh, and I think for better or for worse, I'm really proud of the work that my team has done. And I'm not so sure that other teams in our, in our company do this, but we try to focus pretty heavily on, uh, on skill exposure in, in, in building, in slack time for people to play with ideas and play with projects and, and try to get their grasp around some of these new technologies, even if they go nowhere. Right. I mean, the goal is, is really, I think that when you activate your brain learning something new, it translates no matter what the tool is, right. You've just got to keep that continuous learning model going for yourself personally. And in order for your, for your engineers and for your, for your people to keep doing that, you got to give them time and encouragement to, to do it right. Because otherwise there's always plenty of work to do. Um,
No doubt the work doesn't go away. You
Know, I think, uh, you know, like, you know, I read a lot of like, you know, obviously we're at, we're at the DevOps enterprise summit, so this is, you know, gene Kim's work. And a lot of his, a lot of his stuff, he focuses a lot on that whole concept of improving work, right? You need to him, you need to spend as much time figuring out how you're going to improve your work situation as you do actually doing the work itself. Um, and I'm a big believer in that. The, I will tell you that I have found that people are wired differently in terms of how they embrace that. You know, I have, I have, I have people that are thinkers and people that are doers, and I'm not trying to broadly say that that's the only category that you fit in. I just think that, that, you know, we have tendencies, uh, uh, to, to start with one or the other, utilizing the situation where you're going to start picking up and start running and running and doing it.
Let me, let me understand every single thing. And then I'll start coding or MI, which is
Jane,
Why isn't this working? Why isn't this working? Yeah. I don't understand a darn thing that I'm doing, but, but I'm doing it
And both were fun. And they're pro and once you have to do to, to kind of get the harmony out of your team is play to the strengths of people, but still encourage them to, to, to invest that time and try to think about how you're going to rearrange your workload to make it easier for yourself. Because I have, I have people on my team that are great with checklists, right. You know, you can have a checklist and they go, boom, boom, boom, boom. And then they're done for the day, right? This is, I'm fine with that. And, and that's okay. But the problem is, is if they're doing the same checklist over and over and over again, then they're not growing as a person they're not growing as a, as a, as a team member. And, um, so I have to say, okay, one of your items on your checklist is to take an hour and, you know, learn something new.
Right. You know, go, go, go code something, right. And, and making an actual checklist item. And then I have other people that are much better abstract thinkers. And that's what they do out of the box. Naturally ensure the little to-do items are hard for them. Uh, those folks I love cause, um, that to me, right. You know, I, you know, those are my people and, uh, I'm great at sitting down and thinking through all these great ideas, but when it comes to actually, you know, getting them done on a day to day basis, it's, it's hard for me. And I think, I think lazy people are good at that because we're immediately going to figure out how, how can I do this once and be done with it sometimes to our detriment. But when it comes to, I guess, to steer us back on the course, when it comes to learning new tools and learning new technologies and things like that, I think you have to set, you have to get everybody to the table. You have to make it part of everybody's job to, you know, find the time to do it. But you also have to work within the constraints of what makes them happy and productive. Right. Because I will tell you that if you try to force a, a person that is geared to doing checklists to say, okay, go more than this. And you give them no direction whatsoever. It will fail every time because they, they have no schema for wrapping that up and trying to figure that out. And then on the other
Motivation just doesn't work the same.
Right. Right. And if you, on the other hand, if you take an abstract thinker and say, here, they'll push this button, push this button, push this button, they're going to hate it and they're not going to learn it. So it's, you have to, you have to figure out, particularly in a management role, how do you balance that and how to, how to get that now I'm like I said, I'm very proud of how my team does it. I think that on other teams, particularly when we've had, uh, we've had struggle points because we're so ready to do something new and exciting that, um, you know, it's tough sometimes to get people to drag along and come with us. So we've had to go back and frame the conversation in terms of what is it, what does this make easier for you? Yeah.
Okay. So that brings up a great question. So, so here's, here's the, um, you know, here's this person sitting in the audience right now, right? They're they're, they're a D D E S and they're going like, oh, I'm in. And, but now they've got to go back to their organization and convince people, right? We've got about, we've got about two minutes left, tell them everything they need to know to convince the org to, to adopt this.
The first thing is start with a pilot project, pick a small database, and you decide which I really advocate. I really liked the two-pronged approach of saying, this is the legacy one. This is a new one. Right. And because there are challenges with both of them, and you're only going to learn them if you, if you do both, right, because, uh, you got to do the brownfield and the Greenfield development to do, but start with the pilot projects, start small, start learning how to get your code in and out of a source control and show people how to do it. You got to make it repeatable. And that's the other problem is, is that you can't just because you can do it now, you've got to go and figure about how you're going to go back and tell somebody else about it. Do so, because, uh, you know, just because you've mastered the skill doesn't mean that someone else is going to master it the same way you do.
So I re we started doing, we, we started doing monthly forums where we get together with other teams and, and, and show, right. These are the pilot projects we're working on. Where are you guys working on? Oh, that's interesting. Well, let's go back and bounce those ideas off. Once you get into the source control, do the builds, start looking at continuous integration tools and, uh, start talking about how do you begin to push the button and make it go. I will tell you that you do it one time. And that rush of adrenaline is so awesome. You're like, man, this is great. And then you automatically start calling up your buddies and going, Hey, can you come look at this and, and watch, and you push the button in the desert again. And it's, it's, it's a fantastic feeling, but it is, it does take a lot of energy and a lot of time to get it set up, but start small focus on learning one step at a time share as you go. Um, and then, and then it becomes much easier to begin to having these conversations with people and say, now we need to start investing in tooling to make this happen. Right,
Right. Um, I'll add, I'll just add to it. Cause I mean, I've mainly wanted you to talk, but I will add to that topic. And the one thing I would say is this measure your pain. Um, I always talk about that. Measure your pain. And that's what I mean by that is measure the downtime on deployments measure the amount of time it takes to do to a deployment measure. The times that you've horrifically messed up deployments and had to go into recovery, right. Measure your pain. And, and then, and then that way you've got a, a metric to, to, to say, Hey, look, we've reduced our pain. Right.
I think that's great because you know, that is it. I think a lot of people, we have all of these anecdotal stories about how bad things have been figuring out a way to, to make, you know, what I, well, the one thing that I've learned is is that the C level, uh, the upper management, you know, they've got, they don't have time for a story. They want it, they want a summary. So finding a metric and finding a way to say, this is where we were a month ago. This is where we are now. It works on both the good and the bad level, because if you make it, if you show that it's getting worse over time, they're going to be like, oh, we need to intervene. What are your plans for intervening? And that's where you get to tell the story. Uh, if you show how things are getting better over time, you know, it's gotta be really exciting for them to say, wow, that's great. Then you get to tell them the story. Right. You got to have the, those metrics. You got to have that repeatable measurement of pain. I think that's a great expression.
Okay, cool. Well, thanks. Um, well we gotta wrap it up, um, where we are at our time. So thank you, Stewart. Um, it's been great. Stuart Ainsworth. Um, my name's grant. Frichy appreciate everything that you guys are doing. Hope DOE DOE is going well for you. And, um, that's it.
Thanks. Y'all.