Your Data Nerd Friends Need You

Hello, everyone. My name is Christopher Bergh, and you're going to hear about a topic called "Your data nerd friends need you," and how the world of data analytics in science and engineering is failing, and how the principles from Agile and DevOps and Lean are the way forward.

So let's start off with a really broad separation. So in most of this DevOps conference excuse me, we're talking about transactional systems that take user input or machine input, persist it in the database, and they're built and run according to DevOps principles. But there's a whole other class of systems, analytic systems, that take data from transactional systems and try to analyze it, and they're kind of stuck in the Dark Ages. And so that's what this talk is about. It's really about those analytic systems and the people that operate them. And we're going to talk about that problem, and also talk about the people and how having empathy for them can help, and also talk about the problems, the suffering that they're going through, and what can be done, and what you can do to help.

So everyone's heard about big data or machine learning or AI. It's everywhere. Some people call it the new oil. And it's a huge market, a $189 billion market. And companies are getting acquired for huge amounts of data, 15.7 billion for Tableau, which is a reporting tool, whereas GitHub got bought for 7.5 billion. There are tens of millions of people creating insight from data. I think there's even more of them than software developers. Some people argue that it's 1 in 25 workers are actually somewhat involved in doing data analytics. And it's also a system that's rife with failure. Most projects that deal with data and analytics fail. Gartner estimates 60% to 85% of projects fail. Most organizations who want to be data-driven, that is, at the highest level, kind of like when Gartner says companies want to be digital, they also say companies want to be data-driven. And self-reporting is going down. And most data science projects, where the work is done algorithmically, also fail to get into production. So it's not going well. And so what is that about? Well, in a lot of ways, the data and analytic industry is like the US auto industry in the '70s. They're producing Pacers instead of Corollas. And so the cars that they produce tend to be pretty buggy. There's a lot of errors. It tends to, when they want to change model or get a new model in, it takes weeks or months to deploy. And actually, a lot of data and analytic teams are suffering, and the rate at which people are leaving the field is growing. And so if we look at data and analytics today, and we take just four characteristics of these systems that people do data and analytics, the cycle time at which they can get something from the fingertips of their data scientist or data engineer into production, how many errors they have, how many error-free days they have, how well they collaborate, how well they measure, it's all pretty low. And we actually did a survey last year with Eckerson, and they found the same thing, that the companies are pretty slow to deploy, have way too many errors, and a host of other challenges.

And so let's go back to the beginning and talk about these people who do data and analytics. And so all of us who've-- I've had the fortune of spending half my career in building transactional systems in various forms at various organizations, as an individual contributor and a manager, and then another half building these analytic systems and managing people who did it and doing the work myself. And the people who build and run these analytic systems, they just took a different door than you. They sort of got their BS in computer science and went right instead of went left. And they have lots of different roles. Some people are called data analysts. Some people are called data engineers who build and transform data. Some people are called business analysts, or analysts themselves who do visualizations. There's a role of a data scientist who actually does algorithms. There's managers, there's statisticians, there's database administrators. There's a whole bunch of people who have different roles in trying to work with data. And so if we look at it in some ways, all the systems that you hear about in the DevOps Enterprise Summit are, broadly speaking, transactional systems, CRM, ERP, supply chain, a website, a financial system, an HR system. And all those sources, and then some more, actually get taken by these people and built into an analytical system and pushed out to customers. And so that's what our topic is really about, these analytical systems.

And so to build them, the people who work on it, they work in teams. And they've got just different roles and different people who work. And so there's a data engineer, like I said, who actually is involved in accessing and transforming data. There's a data science person who applies algorithms like segmentation to data. There are self-service or analyst teams who actually visualize data. And then there's a data governance team who tries to catalog and make sure that they reduce the risk of data. And so these teams themselves just have a huge fragmented tool chain. And so every task involved in data and analytics has, broadly speaking, a tool to help them do visualization, help them do algorithms and data science, tools to store data in a database, tools to transform data, ETL, extract, transform, load, ELT tools. There's tools to actually keep track of data in human-readable form, a data catalog, and in every part, there's 50 different tools. Some of them are more code-driven, some of them are more low-code driven, but there's a huge market that's billions and billions of dollars every year, and people love their tools, and they have their favorite tools. And some of them actually may be familiar to people here, like Python. And some of them may not be. And so the teams themselves actually work together. So let's say, let's just go through a simple scenario. So there's a data engineer. He or she sources the data from somewhere, loads it into a table in a database. Right? And it's a sales table. It's got Joe and Kelly and a name and a sales. And then there's a data science team who takes that and actually puts an algorithm on top of it. Maybe they cluster it and say, "Who are our high-value customers and low-value customers?" And after that, there's a self-service team. And maybe when I mean self-service, it turns out in data and analytics, sometimes the people who do the nice charts and graphs work for, and we'll talk this, work for the same team, and sometimes they don't. But they're basically putting charts and graphs on it, but they also can add more data into it. Maybe an owner, and there's a whole set of tools for self-service data prep. And then finally, at the end, the poor data governance people is trying to keep track of, well, where did the name come from? Where did sales come from? Where did the segment come from? Who's the owner? What's the lineage? And so they work together, and they also source data, like I said, from all this diversity of systems at different frequencies. Some of it's big, some of it's small, some of it's fast, some of it's slow, some of it's structured, some of it's unstructured. But they take all that and put it into what I think of as a factory. They assemble artifacts of data, and they have different steps along that factory where they put data somewhere, they group it together, they model it, they visualize it, they catalog it, and step by step, data is transformed, artifacts are created, like charts and graphs from data, and finally it gets to the customer. And so it's a lot like a factory, but it's also a lot like a developer system, where because they've got developer boxes and developer systems that need to be deployed continuously from dev to production. And so

they've got this really contradictory set of challenges, and I felt this when I led data and analytic teams. One is you just never like to hear about data problems from your customer, because they call you up, they yell at you, they are unhappy, and then they end up not trusting the data, and that's the whole point of business people, is to trust the data so they can make data-driven decisions. And so the other part is you've got to be able to get ideas from your cool data scientists or your data engineers from their brain into production to get feedback. And so both these things are for a lot of data and analytic teams, very opposite. They run in fear mode, where things are going to break, and they've gotten yelled at too many times, or they run in heroism mode. And so these teams themselves, they may actually work for the same boss, and a lot of you here may roll in a DevOps summit, may work for the CIO, the chief information officer in your organization. There's a role that's parallel to that, sometimes works for the CIO, sometimes not, called the chief data officer or chief data and analytics officer. And sometimes all these people work for that person, and he or she may have the sort of full stack from accessing the data all the way to getting value. But sometimes they don't. Sometimes the data warehouse still works for the CIO, and data governance still works for the CIO. There may be a data science team somewhere off in the organization reporting to the CEO. And then there may be lots of different self-service teams embedded in different lines of businesses, all providing insight. And so this complicated organizational structure's there. And so the idea of a dev relationship and an ops relationship, development and ops is nicely one-to-one in software, but it's pretty many-to-many in,

to use a database term, many-to-many, a DevOps relationship. And let me talk about what that means. So I'm going to go through a couple of slides. So I'm going to go through four columns here, and the first column is that little graphic, is a D in the middle, is a development team, and maybe there's someone doing data engineering on that team, and data science, and they all sort of work together, and maybe it's harder because they're remote employees, or maybe they sit next to each other, and they've got to work together without conflicts. And the software industry has solved this with Git and version control, and yet version control really doesn't have a huge amount of penetration in most data science and engineering projects. Probably maybe a third of them or less than a third actually use version control. And the development team has got some way in a bigger organization to push their work to production. And they need to deploy their work to production so it's safe and they can get feedback and actually get value to the customer. That's fine. That's sort of a DevOps relationship. What's interesting about data and analytics is that there's a lot of self-service teams, and that's these sort of light green Ds here. They have decentralized development. And this happens quite a bit, and actually, most organizations, big organizations, are like this. So you'll have a central development team, like I said, maybe doing a data lake, a data warehouse, a data enablement team. And then every group in the company will have someone with Tableau or Looker, someone maybe writing some Python on top of that data set or around that data set. And so there's this kind of mesh or work of people doing development. And the challenge is the end customer sees the combination of maybe that person using Tableau or Looker, and then the data warehouse team's data all together. And so it gets challenging also when the push to production is also complicated. So you may be able to go in and push a button in Tableau to deploy to Tableau online, but in a centralized development team, you may have to go through several stages, a dev, a QA, a prod system, to get to production. And so you've got this collaboration complexity in data and analytics. That is actually much harder than in software development, because you've got to account for this local and global, centralized, decentralized aspect, multiple ways to get to prod. And it makes it really hard when the CEO says, "Something's screwy in my dashboard." And, "Well, which team did it? And where do they live? And which dashboard is by which person?" And okay, that dashboard was done in the line of business, and oh yeah, the calculation was done by a data science team, but the data actually came from a third group. It came from the data warehouse team, and then the data came from IT somewhere. And so, at the end of the line, being able to fix this and fix it quickly makes it really hard to be a part of these teams. And so,

when I was a software engineer, when we were developing code, we'd sometimes talk about Conway's Law and say, "Oh, Conway's Law sucks. It's bad." If you're developing something in Conway's Law, you're not following the natural way tech wants to go together to be the optimal solution. You're putting partitions in your technology that are based upon the way some organization decided, some VPs in a meeting decided people should work. And I think data analytics follows that quite a bit, that how we partition the value chain that takes data and goes to insight very much follows the Conway's Law. And these pipelines, or more specifically, meta pipelines, the pipeline for a data warehouse and the pipeline for a visualization, pipeline for a model, all are sort of broken up according to Conway's Law. And that actually just makes it really, really hard for teams to work together.

And so that becomes this point, that a lot of data and analytic teams are suffering. They're not having a great time of it. And so, Gene, last year, wrote this book called "The Unicorn Project," and in there, he talked a lot about transactional systems. But he had one chapter where he focused in on these analytical systems, and talked a lot about the five ideals, but also talked about the sort of hero culture or fear culture that goes in in these data teams. Because a lot of them are, in some ways, wear the sort of hair shirt of pain, okay, I'm getting crappy data from someone. My customers don't trust it. I've got to work on Saturdays to fix it. And so you end up with accepting the fact that they've got really high error rates, or that you've got to have a fear culture or a hero culture. And also, a lot of these teams just don't have any automated tests. In fact, most organizations barely even monitor the data as it's flowing through their analytical systems to see if it's even ballpark correct. And so there's a lot of technology review boards to be able to get things out from production. And so average companies are taking weeks or months to deploy, not continuous deployment or continuous integration, just manual step-by-step with a spreadsheet deployment. And so, a lot of companies want to be data-driven. They're fearful of the big Silicon Valley guys, or they've gone to the Gartner conference and have heard about it, and the idea of being data-driven is really important. And the team that helps support a company being data-driven, the data science and data analytics team, are not doing well. They're not being successful. And so, I've experienced it personally myself in managing teams and having customers, like I said, challenge me when things are late, employees wanting to try out new tools, just being able to be desperate to try to get some insight into my hands of my customer.

And so, about three or four years ago, this term DataOps, which is kind of intentionally like DevOps, started to have a bit of a moment. And my company itself has been around for about six years, and being engineers, we weren't really good at naming, and we started to talk about what we did as agile analytic operations, or DevOps for data science, or one guy, we tried Agile DataOps for a while, and that didn't work. And then we tried Analytic Ops, and that didn't work either, because if you shorten it down, it's not the best word. So the idea of DataOps is kind of gathering momentum, and one, from this company, Gartner, put it on its hype cycle. Two, the number of search requests are way up, 500% over two years. We wrote a manifesto that has over 10,000 signatures. And people are starting to talk about it and use that term more. And I think it is really, across all of them, and like any term in tech, it's being used in different ways, is this sort of definition of how you adapt an agile and lean and DevOps mindset to the world of data and analytics.

And so from a definitional standpoint, DataOps is a set of technical practices and cultural norms and architecture patterns that enable, the first thing is really rapid experimentation and innovation. So to give new insights to your customers, you need to be able to do it rapidly and try things out. Because data and analytics, whether it's a data science model or visualization or just data, is a river of questions, and these teams are constantly trying to paddle down this river to find the place where they can give a unique piece of insight to their customer. And so rapid experimentation, but low error rates, where the data's wrong or the data's late, and collaboration across that complicated set of dev and ops, that many-to-many relationship, and the complicated tool chains that they have across cloud and on-prem and the different tools. And then finally, just measuring. How do you measure a system? And so

you may, in your organization, have teams who are starting to hear about DataOps. And, of course, if you're here, you've already heard of DevOps, and your CIO may be saying, "We're going to be agile and we're going to do DevOps." And when you walk across the hall to the data science or the data engineering or the warehouse team, they're like, "Ah, how do we do that?" And so we've actually written quite a bit about this sort of differences between DevOps and DataOps, and I think at a high level, they're kind of the same because they're all sort of, how do you have small batch size, a safe culture? How does everyone deal with this technically complicated thing where we're all sort of touching a piece of the elephant? And so I think one of the main differences is this many-to-many relationship between people who are doing development and people who are operations, which are more complicated in data and analytics. And then this sort of factory method, statistical process control, manufacturing of data sets, I think are just much more prevalent because the manufacturing of insight from all those transactional systems is something that every data and analytic team does. And so what we found is that people who adopt DataOps, they tend to focus on similar things that you would think that data science or that software teams focus on. And the main idea is that there's a system at which they can work, where they can work in a better way, and the characteristics of that system is that it allows them to deploy quicker. And in data and analytics world, it doesn't mean CI or even CD. It means instead of going from three months, they can get down to three days. That's a miraculous change in a lot of organizations. Not let alone from three days to three seconds. And then just trying to run with low errors, and so many companies nowadays have no idea if the analytics they put out is right. They have no idea when they deploy something to production, if it's going to actually work. And the teams themselves are just really frustrated and really beaten down. And I think that's one of my emotional motivations to talk about this to a DevOps audience, is I think there's an opportunity for the skills that you have from DevOps to help these teams.

And I think there's a lot of cases of best-in-class people can do it all with DataOps. They can have fast cycle time and deployment, and as a result, have lower costs and less unhappy customers. And so, well, what can you do to help them? And so I guess the first thing is that you already have, by being in this conference, have a set of ideas around DevOps and what it means to work in an agile way, what it means to iterate, what infrastructure as code is, what having scripts is, what having a safe culture is, how to work through bottlenecks. And those principles themselves, as being a respectful, powerful set of principles that can influence how a team works, they need advocates in data science and analytic teams. Because the mental model they have, it's all about, "I got to do the next chart. I got to do the next model. I got to do the next data set, the next table." It's all very focused on the features that they want to get out, and not the system that they want to build. And if anything, the DevOps idea is that you can build a system in which to do work where people end up being able to do more work that's better. And so that system-wide idea, I think, is important. And so, when you go back to your company, if you're having lunch with someone on a data science team or in data engineering or a data warehouse, ask them some impertinent questions. Like, are you using source control for your work? Simple question. Is your model in Git? Is your visualization in Git? How many automated tests do you have in production? Are you observing the data as it goes through your production systems to see if it's right? Because they're, in some ways, your customer. These operational systems that you guys run in a DevOps world, your CRM system, your website, the tables, the logs that come out of that are actually the input into an analytic process. And so, how do you know if they haven't gotten crappy data they need to automate and test? And then ask another questions. Do they have regression tests, functional tests, or unit tests? Do they just have unit tests? There's some companies who they have a couple of unit tests on their ETL, and then they've plugged some stuff into Jenkins, and they think they're doing miraculous. And so how long does it take to deploy a model or some ETL code from dev to production? Is it automated or is it a manual process? How up-to-date is your development environment? Or how often are your business users finding errors in the data? And every one of you who listens to this, I think knows what kind of answers you want, right? You want fast deployment. You want low errors. You want to be able to have a new person come into your group, get a development environment, be able to fix a simple bug, and deploy to production their first week. Most data and analytic organizations have answers to this that would shock you, that would almost make you want to tear your hair out. And so I think asking these impertinent questions can give you some motivation to say, "Hey, these DevOps principles, these Ops principles, or DataOps principles, or whatever you want to call them, are a way to help these teams."

And so we have, as DataKitchen, we've written a book about DataOps, which is free. We've written a manifesto. And actually on our website, Gene Kim was kind enough last year to let us give out one chapter excerpt of "The Unicorn Project" that talks around Project Panther and the data warehouse team and the challenges that they solved. And so we've got some software that can help with it too. But I just want to thank you and say there is an opportunity for anyone who does DevOps to help out on their data and analytic team. I think it's a growing field. It's got some great trends behind it, and it could use your help. So thank you very much, and have a great day.